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Abstract 

The  Air  Force  is  moving  to  a  network  centric  environment  where  information 
must  be  visible,  accessible,  and  understandable.  This  transformation  has  seen  the 
adoption  of  the  Internet  web  browser  as  a  de  facto  standard  for  information  access.  The 
Technology  Acceptance  Model  suggests  that  information  systems  must  not  only  be  useful 
but  also  be  usable  and  a  large  body  of  usability  engineering  knowledge  exists  to  support 
usable  design.  In  addition,  the  U.S.  government  mandates  specific  minimum  design 
features  required  to  support  disabled  user  access. 

This  research  effort  seeks  to  establish  an  understanding  of  how  well  common 
practice  usability  design  principles  and  government  mandated  accessibility  guidelines  are 
followed  by  Air  Force  intranet  web  sites.  Heuristic  evaluation  is  used  to  investigate  web 
site  usability.  Accessibility  is  inspected  against  government  guidelines. 

The  results  of  this  study  suggest  that  Air  Force  intranet  web  sites  do  not 
adequately  comply  with  many  usability  principles  and  that  accessibility  compliance 
varies  from  site  to  site.  Furthermore,  although  the  majority  of  usability  and  accessibility 
design  principles  are  not  captured  in  military  guidance,  scores  were  higher  for  those 
principles  that  are  captured  in  the  guidance  than  for  those  that  are  not. 
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USABILITY  AND  ACCESSIBILITY  OF  AIR  FORCE  INTRANET  WEB  SITES 


I.  Introduction 

As  the  Department  of  Defense  (DoD)  strives  to  transform  itself  from  the 
Industrial  Age  to  the  Information  Age,  Internet  based  technologies  have  become  a  central 
enabler  of  the  transformation  process.  DoD  policy  recognizes  that  “Data  is  an  essential 
enabler  of  network-centric  warfare  (NCW)”  and  mandates  that  data  “shall  be  made 
visible,  accessible,  and  understandable  to  any  potential  user  in  the  Department  of  Defense 
as  early  as  possible  in  the  life  cycle  to  support  mission  objectives”  (Department  of 
Defense,  2004).  Numerous  enterprise  and  information  architecture  efforts  are  underway 
to  ensure  that  data  is  technically  accessible  and  usable.  The  Air  Force  recognizes  the 
Internet  as  “an  indispensable  source  for  information”  (Department  of  the  Air  Force, 
2005b)  and  has  implemented  many  web  enabled  solutions. 

However,  making  the  data  technically  accessible  and  usable  is  often  not  enough  to 
ensure  a  system’s  success.  Particular  attention  must  be  paid  to  the  human  interface  that 
allows  the  user  to  interact  with  the  data.  Technology  solutions  that  users  find  difficult  to 
use  may  be  under-utilized  or  rejected  outright.  Interestingly,  past  research  conducted  at 
the  Air  Force  Institute  of  Technology  (AFIT)  found  that  usability  and  disabled  access  of 
evaluated  web  sites  were  far  from  optimal  (Felax,  2005;  Kastenholz,  2005).  This  thesis 
explores  Air  Force  intranet  web  sites  with  the  goal  of  creating  a  better  understanding  of 
the  current  state  of  affairs  of  the  usability  and  accessibility  of  those  sites. 
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Problem  Statement 


As  use  of  the  World  Wide  Web  (WWW)  has  increased  and  approaches  the  point 
of  critical  mass,  the  DoD,  recognizing  the  potential  benefits  of  embracing  this 
technology,  began  to  “strongly  encourage”  WWW  use  (Department  of  Defense,  2001). 
Today,  the  Air  Force  portal  boasts  over  700,000  U.S.  military,  civil  service,  ally,  and 
contractor  users  (Department  of  the  Air  Force,  2006).  However,  the  question  of  whether 
users  will  accept  web  based  applications  and  integrate  them  into  their  daily  processes 
remains  unknown  at  this  point  in  time.  While  infonnation  is  a  key  instrument  of  national 
power  and  exploiting  infonnation  remains  a  tenet  of  Joint  doctrine  (Department  of 
Defense,  2005a),  care  must  be  taken  to  ensure  that  infonnation  and  infonnation  systems 
are  usable  and  actually  reduce  rather  than  contribute  to  the  fog  and  friction  of  war¬ 
fighting  operations. 

This  research  expands  upon  previous  efforts  that  found  that  the  usability  and 
accessibility  of  two  Air  Force  web  based  applications  failed  to  meet  many  common 
practice  guidelines  used  in  the  civilian  world.  A  range  of  Air  Force  web  sites  are 
investigated  for  usability  and  accessibility.  Meanwhile,  DoD  and  Air  Force  guidance  are 
reviewed  to  detennine  the  adequacy  of  current  guidance  for  producing  usable  and 
accessible  web  sites. 

Background 

DoD  transformation  efforts  address  three  major  areas:  conducting  business  within 
the  Department,  working  with  “interagency  and  multinational  partners,”  and  how  the 
DoD  fights  (Department  of  Defense,  2005b).  Network-centric  Warfare  (NCW)  is 
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primarily  directed  at  how  it  tights,  but  touches  all  three  areas.  Data,  as  an  enabler  of 
NCW,  must  be  made  readily  available  for  machine-to-machine,  human-to-human,  and 
human-to-machine  interfaces.  Where  human-to-machine  interfaces  occur,  human- 
computer  interaction  (HCI)  theories  and  the  technology  acceptance  model  (TAM) 
provide  valuable  insight  into  the  potential  acceptability  of  these  interfaces. 

The  technology  acceptance  model  suggests  that  perceived  usefulness  and 
perceived  ease  of  use  are  significant  determinants  of  user  acceptance  of  computer 
technologies  (Davis,  1989).  Usability  and  disabled  accessibility  both  play  significant 
roles  in  user’s  perceived  ease  of  use.  In  fact,  some  definitions  of  usability  are 
synonymous  with  ease  of  use  (Bevan,  1995). 

Comparing  Air  Force  web  sites  with  the  most  commonly  accepted  usability 
principles  is  a  practical  way  to  evaluate  the  usability  of  those  sites.  Similarly,  evaluating 
Air  Force  web  sites  against  the  federal  government’s  mandatory  accessibility  guidelines 
provides  an  understanding  of  how  accessible  those  sites  really  are.  Additionally,  a 
review  of  DoD  and  Air  Force  guidance  is  warranted  to  examine  how  well  common 
practice  guidelines  have  been  integrated  into  organizational  policies  and  what  impact  this 
guidance  may  have  on  the  usability  and  accessibility  of  Air  Force  intranet  web  sites. 

Research  and  Investigative  Questions 

This  research  seeks  to  answer  the  question:  How  well  are  appropriate  common 
practice  web  usability  principles  and  Section  508  accessibility  guidelines  followed  by  Air 
Force  intranet  web  sites?  In  order  to  answer  this  research  question,  several  investigative 
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questions  are  used  to  narrow  the  research  effort  to  a  manageable  scope.  These 
investigative  questions  are: 

IQ  1 .  How  well  do  Air  Force  intranet  web  sites  follow  common  practice 
usability  principles  identified  in  the  literature? 

IQ2.  How  well  do  Air  Force  intranet  web  sites  adhere  to  the  accessibility 
requirements  of  Section  508  of  the  Rehabilitation  Act? 

IQ3.  Does  a  relationship  exist  between  usability  and  Section  508  accessibility 
compliance  scores? 

IQ4.  Are  common  practice  usability  principles  and  government  mandated 
Section  508  accessibility  requirements  reflected  in  or  referenced  by 
military  guidance  that  is  relevant  to  Air  Force  intranet  web  sites? 

IQ5.  Does  a  relationship  exist  between  compliance  scores  and  the  existence  of 
common  practice  usability  principles  and  Section  508  accessibility 
guidelines  in  military  guidance? 

Proposed  Methodology 

This  research  implements  a  case  study  approach.  Nielsen’s  usability  heuristics 
(Nielsen  and  Mack,  1994)  are  used  to  inspect  usability  for  the  sites  in  this  study  to  answer 
IQ  1 .  Each  web  site  is  then  evaluated  using  a  checklist  of  Section  508  mandatory 
compliance  items  to  answer  IQ2.  Descriptive  statistics  are  utilized  to  establish  whether  a 
relationship  exists  between  the  usability  and  accessibility  compliance  scores  to  answer 
IQ3.  Next,  IQ4  is  addressed  through  a  review  of  DoD  and  Air  Force  guidance  relevant  to 
web  site  design  and  maintenance.  Lastly,  descriptive  statistics  are  utilized  to  establish  the 
strength  of  the  relationship  between  existence  of  usability  and  accessibility  principles  in 
military  guidance  documents  and  the  respective  compliance  scores  for  those  principles  to 
answer  IQ  5. 
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Significance 

The  Department  of  Defense  and  the  Air  Force  are  making  huge  investments  in  the 
shift  to  NCW.  Portals  and  other  web  sites  provide  the  primary  human-computer  interface 
to  access  data  in  this  new  paradigm.  For  NCW  to  function  at  the  most  efficient  level 
possible,  it  is  essential  that  these  interfaces  meet  user  expectations  for  usability  and  are 
accessible  to  all  users.  Otherwise  users  may  fail  to  fully  accept  these  new  web-based 
tools  and  are  likely  to  rely  instead  on  the  old  ways  of  doing  business  and  develop 
workarounds  for  the  newer  systems. 

Past  research  efforts  found  that  specific  Air  Force  web  applications  failed  to  meet 
reasonable  standards  of  usability  and  accessibility  (Felax,  2005;  Kastenholz,  2005).  The 
current  research  investigates  a  variety  of  additional  Air  Force  web  sites  to  determine  if 
usability  and  accessibility  shortcomings  may  be  prevalent  on  a  wide  scale.  Specific  focus 
is  given  to  the  web  development  and  maintenance  guidance  provided  by  the  DoD  and  Air 
Force  to  detennine  if  that  guidance  is  sufficient  to  guide  the  development  of  usable  and 
accessible  web  interfaces. 

This  research  generates  a  better  understanding  of  the  current  state  of  usability  and 
accessibility  of  Air  Force  intranet  web  sites.  Furthermore,  the  knowledge  gained  from 
assessing  the  adequacy  of  existing  military  guidance  is  useful  to  policy  makers,  web 
interface  designers,  and  those  who  are  responsible  for  acquiring  web-enabled 
technologies. 
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Scope 


Past  research  efforts  have  focused  on  specific  web  sites  and  web-based  tools 
(Felax,  2005;  Kastenholz,  2005).  This  study  includes  six  Air  Force  intranet  web  sites 
whose  purpose  is  to  facilitate  a  transaction  or  an  information  exchange  between  a  host 
organization  and  an  authorized  user  for  a  business  purpose,  other  than  those  with  the 
general  public.  Although  these  systems  are  available  worldwide,  they  are  only  accessible 
to  military  members  and  organizational  partners.  Therefore,  these  systems  are  treated  as 
part  of  a  single  Air  Force  intranet.  Internet  sites  whose  primary  purpose  is  sharing 
information  with  the  public  are  not  included  in  the  scope  of  this  study. 

Limitations 

This  study  attempts  to  overcome  the  primary  limitation  of  past  AFIT  studies  by 
using  multiple  usability  evaluators.  However,  use  of  graduate  students  in  lieu  of 
experienced  usability  experts  may  result  in  less  reliable  scores.  Although  the  scores  may 
vary  from  those  that  would  be  obtained  by  using  experienced  evaluators,  it  is  understood 
that  experts  will  actually  identify  more  shortcomings,  resulting  in  lower  scores  (Nielsen, 
1992).  Additionally,  use  of  a  single  accessibility  evaluator  provides  weaker  support  for 
accessibility  findings  than  would  the  use  of  multiple  evaluators.  However,  accessibility 
evaluations  are  more  technical  in  nature  and  the  additional  evaluators  available  for  this 
effort  did  not  possess  the  technical  skills  required  to  conduct  accessibility  inspections. 

Two  other  limitations  are  the  limited  availability  of  relevant  web  specific  military 
guidance  and  the  unknown  degree  to  which  individual  military  units  comply  with  that 
guidance.  Although  general  guidance  documents  such  as  DoD  Regulations  and  Air  Force 
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Instructions  are  readily  available  for  evaluation,  it  is  nearly  impossible  to  capture  all 
design  documents  used  for  development  and  acquisition  of  web-based  applications.  It  is 
equally  difficult  to  obtain  the  indefinite  quantity  of  organizational  guidance  that  has  been 
produced  for  use  at  the  local  unit  level.  Furthennore,  although  military  guidance  can 
easily  be  qualified  as  mandatory  or  optional,  it  is  not  possible  to  know  with  any  level  of 
certainty  how  well  any  given  unit  complies  with  that  guidance  or  is  even  familiar  with  it 
without  a  unit  compliance  inspection  which  is  beyond  the  scope  of  this  thesis.  Therefore, 
compliance  with  mandatory  guidance  must  be  assumed. 

For  the  purposes  of  this  research,  the  compliance  scores  obtained  by  the 
evaluators  will  be  sufficient  to  indicate  patterns  and  bring  much  needed  attention  to  the 
issues  of  usability  and  accessibility  of  web  interfaces  in  modern,  information  dependent 
network-centric  warfare.  Similarly  a  review  of  available  military  guidance  is  adequate  to 
build  a  general  understanding  of  whether  policy  makers  have  provided  sufficient 
guidance  to  produce  usable  and  accessible  web-based  solutions. 

Thesis  Overview 

The  first  chapter  of  this  thesis  provided  an  introduction  to  the  growing  use  of  web 
interfaces  for  DoD  and  Air  Force  transformation  efforts  and  network-centric  warfare.  It 
also  introduced  the  role  of  usability  and  accessibility  in  technology  acceptance  and  noted 
that  Air  Force  web-based  tools  have  performed  poorly  in  past  usability  and  accessibility 
evaluations,  supporting  the  need  for  the  additional  research.  Chapter  II  contains  a 
literature  review  that  more  fully  supports  the  need  for  usability,  identifies  common 
practice  usability  guidance  applicable  to  web  site  development,  defines  Section  508 
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accessibility  mandates,  and  addresses  specific  usability  and  accessibility  evaluation 
methods.  Chapter  III  details  the  methodology  used  to  answer  the  investigative  questions 
and  describes  the  data  collection  effort.  Data  analysis  is  presented  in  Chapter  IV. 
Chapter  V  contains  additional  discussion  and  recommendations  for  future  research. 
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II.  Literature  Review 


As  the  Air  Force  moves  to  a  web-based  environment,  it  is  important  to  understand 
the  factors  that  are  driving  this  paradigm  shift  and  recognize  potential  impediments  to 
successful  transformation.  This  chapter  begins  with  an  overview  of  Internet  technology 
used  for  electronic  business  (e-business)  and  organizational  intranets  and  discusses 
usability  as  a  detenninant  of  technology  acceptance.  A  set  of  characteristics  that  capture 
the  subjective  concept  of  usability  are  identified  and  various  usability  inspection 
techniques  assessed.  Accessibility  is  established  as  a  mandatory  feature  of  Air  Force 
intranet  web  sites  and  accessibility  inspection  techniques  are  identified.  The  theoretical 
relationship  between  usability  and  accessibility  is  discussed  and  the  role  of  guidance  in 
developing  usable  and  accessible  web  sites  is  explored.  This  chapter  concludes  with  a 
review  of  the  findings  of  past  research  in  this  area  and  presentation  of  a  model  for  the 
current  research. 

Electronic  Business  and  the  World  Wide  Web 

Businesses  have  long  recognized  the  value  of  computer  technologies  for  reducing 
transaction  costs  associated  with  daily  activities.  E-business  integrates  these  technologies 
to  create  new  business  processes  that  facilitate  lower  operating  costs,  better  service,  and 
network  effects  (Van  Slyke  and  Belanger,  2003).  IBM,  an  early  developer  of  e-business 
concepts,  defines  e-business  as  “The  transfonnation  of  key  business  process  through  the 
use  of  Internet  technologies”  (Tracy,  2000).  Academic  definitions  tend  to  coincide  with 
IBM’s  characterization.  For  instance,  according  to  Jones  et  al,  “E-business  is  the 
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carrying  out  of  business  activities  that  lead  to  an  exchange  of  value,  where  the  parties 
interact  electronically,  using  network  or  telecommunications  technologies”  (Jones  et  ah, 
2000).  More  simply  stated,  e-business  “is  the  use  of  electronic  communications  networks 
to  allow  organizations  to  send  and  receive  information”  (Van  Slyke  and  Belanger,  2003). 

E-business  transactions  use  a  variety  of  electronic  communications  techniques, 
many  of  which  are  based  on  Internet  technologies  and  utilize  hypertext  based  interfaces 
originally  developed  for  use  on  the  World  Wide  Web  (WWW).  Although  Tim  Berners- 
Lee  is  credited  with  creating  the  WWW  in  1990,  many  of  the  Internet  technologies  that 
underpin  the  WWW  had  already  been  in  development  for  a  number  of  years.  Among  the 
most  significant  of  these  technologies,  the  concept  of  linking  documents  to  related 
documents  was  first  envisioned  by  Vannevar  Bush  in  1945  and  was  later  coined  hypertext 
by  Ted  Nelson  in  1965.  Although  numerous  independent  projects  implemented  hypertext 
concepts  from  1965  though  the  late  1980s,  the  WWW  was  the  first  to  make  this  powerful 
capability  available  to  anyone  with  Internet  access  (Myers,  1998). 

Hypertext  is  implemented  through  a  markup  language,  the  most  common  of 
which  is  HyperText  Markup  Language  (HTML).  These  markup  languages  describe  how 
the  document  should  be  displayed  by  a  web  browser.  Web  browsers  reside  on  the  client’s 
local  machine  and  are  capable  of  retrieving  and  displaying  a  variety  of  hypertext 
formatted  documents  from  remote  servers.  This  combination  of  browser  technology  and 
HTML  web  documents  provides  the  human-computer  interface  whereby  users  can  access 
the  vast  body  of  information  available  on  the  WWW. 
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Intranets  and  Extranets  in  Organizations 

It  is  important  to  note  that  e-business  is  not  limited  to  the  Internet  but  also  occurs 
on  private  and  semi-private  networks  known  as  intranets  and  extranets.  By  strict 
definition,  intranets  are  networks  open  only  to  organizational  members.  In  contrast, 
extranets  allow  organizational  partners  limited,  controlled  access  to  internal  resources. 
Intranets  and  extranets  may  reside  on  privately  owned  or  leased  data  communications 
circuits,  take  advantage  of  modern  technologies  to  create  a  virtual  private  data 
communication  path  using  the  Internet  as  a  backbone,  or  simply  reside  on  the  Internet 
and  utilize  network  security  technologies  to  limit  access  (FitzGerald  and  Dennis,  2005; 
Van  Slyke  and  Belanger,  2003). 

The  textbook  distinction  between  intranets  and  extranets  is  useful  in  a  learning 
environment  where  each  can  be  assigned  a  clearly  defined  role.  In  practice  however, 
extranets  are  often  simply  extensions  of  organizational  intranets.  Indeed,  the  line 
between  intranets  and  extranets  is  often  blurred  as  organizations  find  increased  value  in 
efficient  communications  with  its  partners  (Emery  et  ah,  2005;  Yen  and  Chou,  2001). 

A  fundamental  role  of  these  intranet/extranets  is  to  serve  as  an  information 
integrator,  drawing  together  information  from  various  functional  areas  and  incompatible 
information  systems  into  a  single  system  that  provides  a  common  interface  to  access 
organizational  resources  (Emery  et  al.,  2005;  Scheepers  and  Damsgaard,  1997).  Intranets 
reduce  costs  and  improve  efficiency  by  providing  an  alternative  to  paper-based 
information  distribution  and  reduce  infrastructure  and  training  costs  by  implementing 
user  interfaces  that  are  compatible  with  existing,  inexpensive  assets.  Applications  such 
as  electronic  mail  (e-mail)  and  web  servers  save  time  by  speeding  up  communications 
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and  allowing  easy  access  to  new  and  updated  information  (Yen  and  Chou,  2001).  Other 
uses  of  intranets  in  the  commercial  sector  include  dissemination  of  organizational 
policies,  workflow  management,  just-in-time  training,  and  multi-point  document 
authoring  (Emery  et  al.,  2005). 

Although  the  purpose  of  intranets  and  extranets  differs  from  the  greater 
functionality  and  reach  of  the  Internet,  many  of  the  same  design  principles  that  apply  to 
Internet  site  design  are  equally  applicable  to  these  smaller  networks.  However,  some 
nuances  exist  such  as  the  user’s  purpose  and  the  hardware  used  to  access  the  network. 
Intranets  and  extranets  should  be  organized  for  employees  and  organizational  partners  to 
gain  quick  access  to  work  related  infonnation.  Meanwhile,  internet  sites  are  typically 
designed  to  make  only  a  limited  amount  of  the  organization’s  public  infonnation 
available  to  a  wide  range  of  external  users.  Another  difference  is  that  intranet  site 
designers  have  more  freedom  to  use  design  solutions  targeted  at  a  known  environment 
where  Internet  and  extranet  site  design  must  provide  technical  accessibility  for  a  wider 
range  of  technologies  (Nielsen,  2000). 

E-business  and  the  Department  of  Defense 

Recognizing  the  success  of  e-business  practices  in  the  commercial  sector,  the 
United  States  government  began  implementing  e-business  policies  in  the  mid-1990s.  The 
DoD  quickly  followed  suit  with  the  Secretary  of  Defense’s  Defense  Reform  Initiative 
Report  in  November  1997  (Department  of  Defense,  1997).  The  report  encouraged  the 
DoD  to  look  beyond  traditional  electronic  commerce  efforts  and  explore  “expanding 
functional  applications  beyond  supply,  procurement  and  accounting  to  enabling  process 
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improvements  in  other  functional  areas  such  as  health,  personnel,  systems  acquisition  and 
science  and  technology”  (Department  of  Defense,  1999).  Today,  DoD  e-business 
initiatives  can  be  found  throughout  all  functional  areas. 

E-business  concepts  leverage  infonnation  technology  to  break  paradigms  and 
create  more  efficient  processes.  These  are  shared  objectives  of  network-centric  warfare, 
a  central  enabler  of  DoD  force  transformation  efforts  (Department  of  Defense,  2005a). 

By  moving  to  a  highly  interconnected  environment,  the  DoD  hopes  to  leverage  the 
resulting  network  effects  to  realize  a  shift  to  a  “more  effective  warlighting  style 
characterized  by  the  new  concepts  of  speed  of  command  and  self-synchronization” 
(Cebrowski  and  Garstka,  1998). 

E-business  concepts  and  network-centric  warfare  infer  a  certain  amount  of 
interoperability  between  processes  and  between  systems.  To  that  end,  the  DoD  has 
established  extensive  guidance  to  ensure  systems  are  interoperable  and  network  ready. 
Relevant  guidance  such  as  DoD  Directive  8100.1  and  Chainnan  of  the  Joint  Chiefs  of 
Staff  Manual  3 170.0  IB  lay  the  foundation  for  force  transformation  and  network-centric 
warfare  and  reiterate  the  importance  of  interoperable  systems  and  accessible  data 
(Department  of  Defense,  2002;  Joint  Chiefs  of  Staff,  2005). 

The  Air  Force  in  the  Web  Environment 

DoD  E-business  and  force  transformation  initiatives  are  present  in  all  DoD 
services.  Within  the  Air  Force  e-business,  formally  named  Electronic 
Business/Electronic  Commerce  (EB/EC),  is  defined  as: 
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1 .  The  interchange  and  processing  of  information  via  electronic  techniques 
for  accomplishing  transactions  based  upon  the  applications  of  commercial 
standards  and  practices. 

2.  The  conducting  of  business  communications  and  transactions  over 
networks  and  through  computers.  As  most  restrictively  defined,  EB/EC  is 
the  buying  and  selling  of  goods  and  services,  and  the  transfer  of  funds, 
through  digital  communications.  EB/EC  also  includes  all  intercompany 
and  intracompany  functions  (such  as  marketing,  finance,  manufacturing, 
selling,  and  negotiation)  that  enable  commerce  and  use  electronic  mail, 
electronic  data  interchange,  file  transfer,  facsimile,  video  conferencing, 
workflow,  or  interaction  with  a  remote  computer  (including  use  of  the 
World  Wide  Web).  (Department  of  the  Air  Force,  2004) 

These  definitions  are  very  similar  to  the  academic  and  practitioner  e-business 

definitions  previously  discussed.  Like  the  commercial  sector,  Air  Force  e-business 

efforts  exist  on  the  Internet  as  well  as  on  intranets  and  extranets.  However,  the  US 

military  is  somewhat  unique  in  that  they  operate  their  own  domain  (.mil)  which  is  part  of 

the  larger  Internet  but  not  all  of  which  is  accessible  to  all  Internet  users.  Some  restricted 

portions  of  the  military  domain  serve  an  intranet  and  are  only  open  to  military  users. 

Other  restricted  portions  also  function  as  an  extranet  and  are  accessible  by  military  users 

and  by  organizational  partners  including  foreign  militaries,  research  institutes,  and 

military  contractors  who  may  be  granted  limited  or  full  access  for  officially  sanctioned 

use. 


Within  the  Air  Force,  “firewall  servers  and  other  security  measures”  are  used  to 
create  a  global  intranet  that  allows  authorized  users  to  share  resources  “without  making 
sensitive  information  available  to  users  with  Internet  access”  (Department  of  the  Air 
Force,  2004).  This  policy  tolerates  the  blurring  of  intranets  and  extranets  commonly 
found  in  the  private  sector  but  clearly  recognizes  the  need  for  separate  private  and  public 
use  networks.  This  research  effort  therefore  defines  an  Air  Force  intranet  web  site  as  one 
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whose  purpose  is  to  facilitate  a  transaction  or  an  information  exchange  between  a  host 
organization  and  an  authorized  user  for  a  business  purpose  other  than  those  with  the 
general  public.  This  working  definition  does  not  include  local  intranets  open  only  to 
local  organizational  users  such  as  wing  or  base  intranets  nor  does  it  include  information 
providing  web  sites  open  to  the  general  Internet  user. 

Air  Force  use  of  the  Internet  and  intranet  technologies  has  seen  an  explosive 
growth  over  the  past  decade.  However,  very  little  research  documenting  this  growth  is 
available.  In  1996,  Quintero  found  that  Air  Force  users  who  she  surveyed  used  Internet 
access  to  “conduct  research,  obtain  answers  to  specific  questions,  obtain  or  access 
regulations,  and  access  their  own  organizational  or  other  organization’s  homepage”  and 
“to  obtain  forms,  weather  reports,  and  travel  infonnation,  as  well  as  conducting  market 
research,  gaining  access  to  Congressional  reports,  and  researching  legal  databases” 
(Quintero,  1996).  By  2001,  Internet  usage  had  become  so  common  that  the  Air  Force 
implemented  the  Air  Force  Portal  web  site  in  an  effort  to  consolidate  access  to  the 
growing  number  of  web  sites  and  applications  used  by  its  members.  As  of  January  2006, 
the  portal  boasted  more  than  700,000  users  and  provided  access  to  over  eighty  tools  and 
applications  (Department  of  the  Air  Force,  2005a;  Department  of  the  Air  Force,  2006). 

With  the  current  emphasis  on  the  use  of  intranets  and  the  Internet,  it  is  important 
to  consider  what  factors  are  necessary  for  a  web  site  to  be  successful.  A  primary  concern 
for  any  information  system  (IS)  is  whether  it  will  be  used  by  organizational  members  and 
Air  Force  intranet  web  sites  are  not  exempt  from  this  concern.  Considerable  research  has 
been  accomplished  in  the  area  of  understanding  the  factors  that  lead  to  information 
system  success.  The  Technology  Acceptance  Model  (Davis,  1989)  is  one  such  model 
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and  its  underlying  concepts  support  the  need  for  information  systems  to  be  usable  and 
accessible. 

Technology  Acceptance 

As  use  of  information  technology  continues  to  expand  and  technological  barriers 
to  use  disappear,  people’s  willingness  to  use  the  infonnation  systems  remains  a  concern 
of  researchers  and  practitioners  alike.  At  present,  there  are  several  theories  and  models 
that  attempt  to  understand  detenninants  of  information  system  adoption  and  predict  user 
acceptance  of  new  computer  technologies.  Some  of  the  most  widely  used  include  the 
technology  acceptance  model  (TAM),  the  theory  of  planned  behavior  (TPB),  and 
innovation  diffusion  theory.  Although  many  of  the  current  adoption  theories  are  widely 
supported,  TAM  has  been  validated  in  a  number  of  environments  (Dasgupta  et  ah,  2002; 
Lederer  et  ah,  2000)  and  its  perceived  ease  of  use  determinant  makes  it  especially 
relevant  to  this  research. 

Although  several  extensions  to  TAM  have  been  proposed,  the  core  model  relies 
on  two  determinates,  perceived  usefulness  (PU)  and  perceived  ease  of  use  (PEOU),  as 
depicted  in  Figure  1 .  Perceived  usefulness  is  defined  as  “the  degree  to  which  a  person 
believes  that  using  a  particular  system  would  enhance  his  or  her  job  performance”  while 
perceived  ease  of  use  is  “the  degree  to  which  a  person  believes  that  using  a  particular 
system  would  be  free  of  effort”  (Davis,  1989).  More  simply  stated,  PU  corresponds  to 
how  helpful  the  user  believes  the  system  to  be  and  PEOU  represents  the  usability  of  the 
system.  These  determinates  have  been  widely  research  and  are  supported  by  an 
“accumulated  body  of  knowledge  regarding  self-efficacy,  contingent  decision  behavior, 
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and  adoption  of  innovation”  (Davis,  1989).  Proponents  of  TAM  theorize  that  these 
determinants  create  a  favorable  attitude  toward  using  the  system  which  positively  affects 
intention  to  use  the  system  (Davis,  1989). 


Figure  1.  Technology  Acceptance  Model  (Davis  et  al.,  1989) 


Traditional  TAM  models  indicate  that  PEOU  operates  on  behavioral  intention  to 
use  through  attitude  toward  using  and  operates  directly  on  PU.  Research  efforts  have 
resulted  in  mixed  findings  on  the  impact  of  PEOU  on  behavioral  intention  to  use  through 
attitude  (Geffen  and  Straub,  2000)  but  repeatedly  confirm  the  affect  of  PEOU  on  PU 
(Dasgupta  et  al.,  2002).  One  possible  explanation  for  these  varied  findings  is  that  new 
users  initially  view  the  system  from  a  self-efficacy  perspective,  detennining  PEOU  based 
on  how  successful  they  expect  to  be  in  using  the  system.  However,  as  they  become  more 
proficient  with  the  system  PEOU  evolves  to  into  an  “instrumental  issue”  of  how  useful 
the  system  is  considering  on  the  relative  amount  of  effort  required  to  use  it  (Davis  et  al 
1989). 

Geffen  and  Straub  offer  another  explanation.  They  suggest  that  the  varying 
findings  of  the  effects  of  PEOU  may  be  related  to  the  nature  of  the  task  to  be  performed. 
Their  research  indicates  that  PEOU  operates  on  behavioral  intention  to  use  when  the  task 
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is  intrinsic  in  nature  and  “the  IT  itself  provides  the  primary  product  or  service”  as  in  the 
case  of  an  information  inquiry  (Geffen  and  Straub,  2000).  In  contrast,  PEOU  may  have 
no  direct  affect  on  behavioral  intention  to  use  when  the  IT  is  simply  a  “means”  as  in  the 
case  of  purchasing  an  item  from  a  web  site  where  PEOU  of  the  web  site  “is  not  an 
inherent  quality  of  the  purchased  product”  (Geffen  and  Straub,  2000).  Taken  together, 
Davis’s  and  Geffen  and  Straub’s  explanations  suggest  that  user  perception  of  a  web  site’s 
usability  will  directly  affect  user  acceptance  through  behavioral  intention  to  use  when  the 
primary  purpose  is  information  sharing  and  have  an  indirect  affect  through  PU  for  both 
information  and  transaction  oriented  web  sites. 


Figure  2.  Geffen  and  Straub's  Research  Model  (Geffen  and  Straub,  2000). 


Two  additional  issues  must  be  addressed  to  support  TAM’s  applicability  to  Air 
Force  intranet  web  sites  and  web-enabled  applications.  The  first  is  issue  whether  TAM  is 
applicable  to  the  Internet  and  Internet-technology  based  intranets.  The  second  issue  is 
whether  TAM  is  applicable  in  the  military’s  mandatory  adoption  environment. 

Regarding  the  first  question,  several  studies  have  found  support  for  TAM  in  web 
environments  (Dasgupta  et  ah,  2002;  Geffen  and  Straub,  2000;  van  der  Heijden,  2003). 
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Even  more  relevant  to  the  current  research  effort,  Horton’s  research  indicates  that  that 
TAM  is  also  applicable  to  Internet-technology  based  intranets  (Horton  et  ah,  2001). 

Rawstome  defines  a  mandatory  adoption  environment  as  one  where  the  “end  user 
is  forced  by  the  organization,  through  reward  inducements  or  threats  of  punishment  or  a 
combination  of  both,  to  utilize  the  IS  in  a  way  that  replaces  at  least  one  previous  work 
practice”  (Rawstorne  et  al.,  1998).  However,  the  applicability  of  TAM  to  any 
environment  presupposes  some  variance  of  adoption  exists.  Hartwick  and  Barki  were 
among  the  first  to  recognize  and  document  that  variance  does  indeed  exist  in  mandatory 
adoption  environments  (Hartwick  and  Barki,  1994).  Rawstorne  also  observed  this 
phenomenon  and  theorized  that  the  existence  of  variance  in  individual  adoption  within  a 
mandatory  adoption  environment  may  stem  from  “(1)  the  extent  to  which  an  individual 
wishes  to  comply  with  the  mandate,  (2)  the  extent  to  which  the  organization  enforces  its 
mandate,  and  (3)  individual  differences  in  using  the  technology”  (Rawstorne  et  ah,  2000). 

Based  on  the  literature,  TAM  is  applicable  to  Air  Force  intranet  web  sites.  This 
suggests  that  user  perceptions  of  a  web  site’s  usefulness  and  usability  are  important 
determinants  for  successful  implementation,  user  adoption,  growth,  and  a  corresponding 
shift  to  e-business  paradigms.  While  both  determinants  are  important,  this  research 
focuses  on  the  usability  aspect  of  Air  Force  intranet  web  sites. 

Usability 

Usability  engineering  is  focused  on  developing  technologies  that  are  useful  and 
usable.  One  facet  of  this  discipline  is  ensuring  that  human-computer  interfaces  maximize 
rather  than  reduce  system  ease  of  use.  Usability  has  an  important  role  in  user  acceptance 
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and  is  especially  important  in  WWW  sites  where  the  first  defense  against  usability 
problems  is  to  leave  and  find  a  more  usable  system  (Nielsen,  2003).  Low  usability  can 
cost  an  organization  in  time,  resources,  and  productivity  and  may  ultimately  cause  a 
system  to  be  rejected  outright.  From  a  user’s  perspective,  high  usability  reduces 
frustration  and  leads  to  more  efficient,  accurate  and  complete  task  accomplishment 
(Foraker  Design,  2005). 

Despite  many  years  of  research,  no  universally  accepted  definition  of  usability  has 
been  developed  (Frokjser  et  al.,  2000).  Some  define  usability  in  terms  of  “quality  of  use” 
(Bevan,  1995)  while  others  focus  on  “ease  of  use”  (Nielsen  and  Phillips,  1993).  Quality 
of  use  definitions  emphasize  the  quality  and  usefulness  of  the  data  provided,  while  ease 
of  use  definitions  focus  on  the  design  of  the  interface  (Bevan,  1995;  Frokjser  et  al.,  2000). 
The  ease  of  use  definition  is  the  most  widely  used  (Bevan,  1995)  and  clearly  links 
usability  to  TAM’s  perceived  ease  of  use  determinant. 

Web  browsers  have  become  a  de  facto  standard  in  e-business  environments  and 

the  same  usability  principles  that  apply  to  traditional  human-computer  interfaces  are 

equally  applicable  to  these  environments  (Brinck  et  al.,  2002;  Mayhew,  2005).  Usable 

web  sites  are  intuitive  and  “allow  users  to  accomplish  their  goals  quickly,  efficiently,  and 

easily”  (Brinck  et  al.,  2002).  Design  goals  for  the  web  include  efficiency,  leamability, 

memorability,  error  tolerance,  and  satisfaction  where: 

Efficiency  refers  to  the  time  or  effort  required  for  users  to  perform  tasks  after 
repeated  exposure  to  the  system  interface. 

Leamability  addresses  the  level  of  difficulty  involved  in  accomplishing  basic 
tasks  during  the  user’s  first  experience  with  the  system. 
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Memorability  determines  the  level  of  difficulty  required  for  infrequent  users 
to  re-learn  the  system  interface  following  periods  of  non-use. 

Error  tolerance  describes  how  well  errors  are  prevented,  the  number  and 
severity  of  errors  a  user  experiences,  and  the  level  of  effort  required  to  recover 
from  errors. 

Satisfaction  is  a  subjective  assessment  of  how  pleasant  the  design  is  to  use. 
(Brinck  et  ah,  2002;  Nielsen,  2003) 

Usability  Inspection  and  Heuristic  Evaluation 

According  to  Zhu,  there  are  at  least  six  primary  classes  of  methods  that  are 
applicable  to  evaluating  web  site  usability.  These  include  usability  inspections,  usability 
testing,  prototyping,  field  methods,  interviews,  and  web-based  methods  (Zhu  et  ah, 

2005).  Usability  inspections  evaluate  the  system  with  usability  experts  and  reliance  on 
the  evaluator’s  judgment  is  their  key  trait  (Nielsen  and  Mack,  1994).  Usability  testing 
most  often  evolves  observing  and  analyzing  user  interaction  with  the  system  in  a 
laboratory  environment.  In  a  prototyping  approach,  a  mock-up  of  the  system  is  used  to 
conduct  usability  inspection  or  testing  to  identify  problems  early  in  development.  Field 
methods  are  similar  to  user  testing  but  differ  in  that  observation  of  users  occurs  in  the 
users’  environment.  Interviews  provide  the  ability  to  assess  user  tasks  and  needs  and  to 
better  understand  the  user’s  subjective  point  of  view.  Web-based  methods  leverage 
Internet  technologies  to  capture  infonnation  about  user  sessions  for  analysis  and  provide 
the  ability  to  solicit  user  feedback  through  questionnaires  (Zhu  et  al.,  2005). 

Among  these  general  methods,  usability  inspection  has  emerged  as  one  of  the 
most  widely-used.  This  is  largely  due  to  its  high  success-rate  for  identifying  usability 
problems  and  low  cost  to  implement  (Nielsen  and  Mack,  1994).  Although  a  variety  of 
inspection  methods  exist,  all  share  the  basic  assumption  that  it  is  possible  to  identify  what 
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features  are  required  for  usability.  Most  inspection  methods  share  two  common 
characteristics.  The  first  is  recognition  that  there  are  many  components  of  usability  and  a 
variety  of  measures  is  required  for  a  full,  accurate  assessment.  The  second  characteristic 
is  that  many  inspection  measures  are  not  intrinsically  objective  and  instead  rely  on 
subjective  assessments  by  the  inspector  (Agarwal  and  Venkatesh,  2002).  Examples  of 
the  various  usability  inspection  methods  are  summarized  in  Table  1. 


Table  1.  Usability  Inspection  Methods  Summarized  (Nielsen  and  Mack,  1994) 


Inspection  Method 

Description 

Cognitive 

Walkthroughs 

Detailed  procedure  aimed  at  determining  whether  the  user’s  problem¬ 
solving  process  will  lead  to  the  next  correct  action 

Consistency 

Inspections 

Designers  of  multiple  systems  inspect  interfaces  to  evaluate 
consistency  across  related  products 

Feature  Inspections 

Focuses  on  the  function  delivered  by  the  system 

Formal  Usability 
Inspections 

A  structured  approach  where  participants  follow  a  six-step  process 
similar  to  software  code  inspections  to  inspect  the  interface  and  create  a 
merged  list  of  problems 

Guideline  Reviews 

Interface  is  checked  for  conformance  against  a  comprehensive  set  of 
usability  guidelines 

Heuristic  Evaluation 

Usability  specialists  judge  interfaces  based  on  general  usability 
principles  called  heuristics 

Pluralistic 

Walkthroughs 

Users,  developers,  and  human  factors  experts  discuss  usability  issues 
based  on  step-by-step  scenarios 

Standard  Inspections 

Interface  is  inspected  against  an  interface  standard  by  an  expert  on  that 
standard  with  the  goal  of  increasing  the  degree  of  compliance  relative 
to  other  interfaces  that  follow  the  same  standards 

One  particular  inspection  method,  heuristic  evaluation,  stands  out  in  web  site 
usability  literature.  In  heuristic  evaluation  “a  number  of  evaluators  are  presented  with  an 
interface  design  and  asked  to  comment  on  it”  (Nielsen  and  Molich,  1990).  Evaluators  are 
provided  with  a  list  of  heuristics  on  which  to  judge  the  system.  These  heuristics  are  more 
“rules  of  thumb”  than  specific  guidelines  (Nielsen  and  Mack,  1994). 
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Heuristics  may  be  so  popular  because  many  web  designers  view  other  usability 
evaluation  methods  as  too  intimidating  or  too  resource  intensive.  Proponents  claim  that 
heuristic  evaluations  cost  less  and  consume  less  time  than  other  methods  and  enjoy  a 
small  learning  curve  that  allows  evaluators  to  become  efficient  quickly  (Nielsen  and 
Mack,  1994).  Comparisons  of  usability  evaluation  methods  have  substantiated  the  value 
of  heuristic  evaluations.  Jefferies  found  that  heuristic  evaluation  identified  more 
problems  and  more  serious  problems  at  lower  cost  than  usability  testing,  guidelines,  or 
cognitive  walkthrough  (Jeffries  et  ah,  1991).  In  another  study,  Nielsen  found  heuristic 
evaluation  would  have  lead  to  the  correct  decision  every  time  if  at  least  three  evaluators 
were  used  (Nielsen  and  Phillips,  1993).  Other  researchers  have  found  that  larger 
numbers  of  evaluators  identify  more  usability  problems,  but  most  agree  that  five 
evaluators  strikes  the  best  balance  of  findings  to  resources  consumed  (Barnum,  2003). 

Nielsen  and  Mack  developed  a  list  of  ten  usability  heuristics  for  interface  design 
that  has  been  widely  used  by  academics  and  practitioners  (Nielsen  and  Mack,  1994). 
Pierotti  developed  a  robust  checklist  to  guide  evaluators  in  applying  these  heuristics  and 
added  three  additional  heuristics  (Pierotti,  1995).  These  thirteen  heuristics  are 
summarized  in  Table  2  and  have  been  previously  detailed  by  a  number  of  researchers 
(Felax,  2005;  Instone,  1997;  Kastenholz,  2005). 

Usability  as  discussed  above  is  driven  by  user  needs  and  preferences.  Traditional 
usability  concerns  are  based  on  a  desire  to  gain  user  acceptance  and  produce  measurable 
efficiencies.  Meeting  usability  standards  is  optional  for  any  organization,  albeit  at  the 
potential  cost  of  lost  users.  However  accessibility  is  federally  mandated  for  government 
entities. 
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Table  2.  Combined  List  of  Nielsen  and  Pierotti’s  Heuristics  (Nielsen  and  Mack,  1994;  Pierotti,  1995) 


Heuristic 

Description 

1.  Visibility  of  System 

Status 

The  system  should  always  keep  users  informed  about  what  is  going  on, 
through  appropriate  feedback  within  reasonable  time. 

2.  Match  Between  System 
and  the  Real  World 

The  system  should  speak  the  users'  language,  with  words,  phrases  and 
concepts  familiar  to  the  user,  rather  than  system-oriented  terms.  Follow 
real-world  conventions,  making  information  appear  in  a  natural  and 
logical  order. 

3.  User  Control  and 

Freedom 

Users  often  choose  system  functions  by  mistake  and  will  need  a  clearly 
marked  "emergency  exit"  to  leave  the  unwanted  state  without  having  to 
go  through  an  extended  dialogue.  Support  undo  and  redo. 

4.  Consistency  and 

Standards 

Users  should  not  have  to  wonder  whether  different  words,  situations, 
or  actions  mean  the  same  thing.  Follow  platform  conventions. 

5.  Flelp  Users  Recognize, 
Diagnose,  and  Recover 
From  Errors 

Error  messages  should  be  expressed  in  plain  language  (no  codes), 
precisely  indicate  the  problem,  and  constructively  suggest  a  solution. 

6.  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which 
prevents  a  problem  from  occurring  in  the  first  place.  Either  eliminate 
error-prone  conditions  or  check  for  them  and  present  users  with  a 
confirmation  option  before  they  commit  to  the  action. 

7.  Recognition  Rather  Than 
Recall 

Minimize  the  user's  memory  load  by  making  objects,  actions,  and 
options  visible.  The  user  should  not  have  to  remember  information 
from  one  part  of  the  dialogue  to  another.  Instructions  for  use  of  the 
system  should  be  visible  or  easily  retrievable  whenever  appropriate. 

8.  Flexibility  and  Efficiency 
of  Use 

Accelerators  —  unseen  by  the  novice  user  —  may  often  speed  up  the 
interaction  for  the  expert  user  such  that  the  system  can  cater  to  both 
inexperienced  and  experienced  users.  Allow  users  to  tailor  frequent 
actions. 

9.  Aesthetic  and  Minimalist 
Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely 
needed.  Every  extra  unit  of  information  in  a  dialogue  competes  with 
the  relevant  units  of  information  and  diminishes  their  relative  visibility. 

10.  Flelp  and  Documentation 

Even  though  it  is  better  if  the  system  can  be  used  without 
documentation,  it  may  be  necessary  to  provide  help  and 
documentation.  Any  such  information  should  be  easy  to  search, 
focused  on  the  user's  task,  list  concrete  steps  to  be  carried  out,  and  not 
be  too  large. 

11.  Skills* 

The  system  should  support,  extend,  supplement,  or  enhance  the  user’s 
skills,  background  knowledge,  and  expertise  -  not  replace  them. 

12.  Pleasurable  and 

Respectful  Interaction 

With  the  User* 

The  user’s  interactions  with  the  system  should  enhance  the  quality  of 
her  or  his  work-life.  The  user  should  be  treated  with  respect.  The 
design  should  be  aesthetically  pleasing  with  artistic  as  well  as 
functional  value. 

13.  Privacy* 

The  system  should  help  the  user  to  protect  personal  or  private 
information  belonging  to  the  user  or  his/her  clients. 

*Added  by  Pierotti 
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Accessibility 

There  are  more  than  50  million  people  in  the  U.S.  alone  who  suffer  from 
disabilities  (Waldrop  and  Stern,  2003).  Many  of  these  disabilities  make  interacting  with 
computers  more  difficult  than  for  non-disabled  people.  Although  military  users  are  not 
traditionally  thought  of  as  including  large  numbers  of  disabled  people,  it  is  important  to 
realize  that  today’s  military  relies  on  a  large  workforce  of  civil  service  employees, 
contractors,  and  civilian  partners  who  play  a  vital  role  in  mission  accomplishment.  Air 
Force  intranet  web  sites  must  be  accessible  to  all  of  these  users  regardless  of  disability. 

Simply  stated,  web  accessibility  means  making  internet  technologies  useable  by 
people  with  disabilities.  Vanherheiden  describes  an  accessible  web  site  as  one  where  the 
content  is  “perceivable,  operable,  and  understandable”  by  any  individual  regardless  of  the 
circumstances  or  constraints  they  are  operating  under  (Vanderheiden,  2005). 

Accessibility  can  be  further  refined  into  technical  accessibility  and  usable  accessibility. 
Technical  accessibility  ensures  the  system  is  compatible  with  assistive  technologies  that 
allow  interaction  with  disabled  people.  For  example,  proper  use  of  HTML  ‘alt’  tags 
allows  associative  technologies  to  convey  the  meaning  of  visual  images  on  web  pages. 
Useable  accessibility  refers  to  the  application  of  basic  usability  principles  for  user 
interfaces.  Examples  include  consistent  navigation  and  management  of  cognitive  load 
(Paddison  and  Englefield,  2004). 

There  are  numerous  reasons  for  web  sites  to  be  accessible.  From  a  humanistic 
perspective,  accessibility  is  the  right  thing  to  do.  This  is  especially  true  when  one 
considers  that  an  underlying  purpose  of  the  WWW  is  to  make  information  available  to 
everyone.  A  second  advantage  is  the  increased  marketability  of  the  site  or  product.  Non- 
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accessible  web  sites  automatically  exclude  disabled  users.  Another  reason  is  that 
accessibility  may  have  spillover  effects  that  benefit  all  users  by  improving  overall 
usability.  In  addition,  many  countries  have  enacted  legislation  that  requires  accessibility 
for  commercial  and  government  web  sites  (Carter  and  Markel,  2001). 

This  last  reason  clearly  impacts  the  DoD.  Section  508  of  the  Rehabilitation  Act 
of  1973  contains  guidance  to  ensure  access  to  the  electronic  and  information  technology 
(EIT)  of  Federal  agencies  by  disabled  persons.  Subsequent  additions  in  1986  and  1998 
further  legislated  access  requirements  and  established  enforcement  provisions  (U.S. 
Access  Board,  2005).  Section  508  mandates  that  all  Federal  departments  and  agencies 
develop  EIT  that  is  accessible  to  “individuals  with  disabilities  who  are  Federal 
employees”  and  “individuals  with  disabilities  who  are  members  of  the  public  seeking 
information  or  services”  unless  doing  so  causes  an  “undue  burden”  on  the  organization 
(Code  of  Federal  Regulations,  2000).  Specific  guidelines  for  web  accessibility  were 
developed  in  2000  and  codified  into  law  in  2001  (Ellison,  2004). 

Although  standards  for  web  compliance  have  been  in  place  since  2001, 
accessibility  compliance  continues  to  be  an  issue  for  some  government  agencies.  In 
2003,  Jackson  found  that  only  60  percent  of  100  government  web  sites  tested  with  an 
automated  tool  passed  evaluation  without  any  critical  problems  (Jackson-Sanbom  et  al., 
2002).  In  a  similar  study  conducted  by  Ellison  in  2004,  44  percent  of  50  government  web 
sites  tested  exhibited  one  or  more  Section  508  compliance  errors  and  only  22  percent 
exhibited  no  errors  (Ellison,  2004).  These  studies  suggest  that  government  agencies  have 
yet  to  meet  the  federally  mandated  rate  of  100  percent  compliance.  It  is  also  interesting 
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that  Jackson’s  study  found  that  the  60  percent  compliance  rate  for  government  web  sites 
was  higher  than  that  of  any  of  the  other  five  types  of  web  sites  included  in  her  study. 

Why  do  so  many  web  sites  remain  inaccessible?  One  reason  is  a  perception  that 
accessibility  is  costly.  However,  experts  believe  that  making  a  site  accessible  adds  only 
one  to  two  percent  to  the  overall  cost  and  is  really  as  simple  as  implementing  HTML  as  it 
was  originally  conceived  (Carter  and  Markel,  2001).  HTML  was  primarily  intended  to 
describe  content  but  its  conventions  are  commonly  misused  as  a  presentation  format. 

This  improper  use  of  HTML  impedes  assistive  technologies  that  rely  on  HTML  tags  to 
provide  a  description  of  the  content.  According  to  the  World  Wide  Web  Consortium 
(W3C),  the  most  common  web  site  accessibility  problems  include  failure  to  provide 
alternate  text  for  images  and  imagemap  hot-spots,  misleading  use  of  structural  elements, 
and  tables  that  are  difficult  to  decipher  when  linearized  (World  Wide  Web  Consortium, 
1999b).  Additionally,  multimedia  content  has  become  commonplace  alongside  the 
primarily  text  content  that  HTML  was  designed  to  support.  Poor  implementation  of  this 
multimedia  content  often  creates  additional  difficulties  for  people  with  disabilities  (Carter 
and  Markel,  2001;  Nielsen,  2000). 

Evaluating  Accessibility  and  Section  508  Compliance 

As  with  usability  evaluations,  there  are  a  number  of  ways  to  detennine  the 
accessibility  of  a  web  site.  Heuristic  evaluations  and  guidelines  are  two  methods  that 
appear  frequently  in  the  literature.  Heuristics  are  recognized  as  a  fast,  easy  method  to 
ascertain  a  site’s  general  accessibility  and  determine  which  problems  are  priorities. 
However,  accessibility  heuristics  tend  to  be  simple  mnemonics  intended  for  use  by 
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accessibility  experts.  These  mnemonics  are  sufficient  to  guide  the  expert  in  his 
evaluation  but  do  not  perform  as  well  when  used  by  non-experts.  Another  drawback  is 
that  although  heuristics  are  able  to  highlight  areas  of  concern  for  disabled  access, 
heuristics  alone  are  not  able  to  substantiate  compliance  with  government  legislation 
(Paddison  and  Englefield,  2004). 

In  contrast  to  heuristics,  guidelines  tend  to  be  bulkier,  more  resource  intensive, 
and  more  difficult  to  remember.  However,  guidelines  are  straightforward  and,  although  a 
level  of  technical  competence  may  be  required,  the  evaluator  does  not  need  to  be  an 
accessibility  expert.  In  fact,  many  software  programs  are  available  to  automatically 
evaluate  web  sites  against  set  guidelines.  Another  strength  of  guidelines  over 
accessibility  heuristics  is  their  ability  to  assess  compliance  with  government  regulation 
(Paddison  and  Englefield,  2004). 

The  Architectural  and  Transportation  Barriers  Compliance  Board,  commonly 
referred  to  as  the  Access  Board,  is  the  source  of  specific,  mandatory  accessibility 
compliance  guidelines  for  federal  agencies.  The  Access  Board  “is  an  independent 
Federal  agency  that  develops  and  maintains  accessibility  requirements,  provides  technical 
assistance  and  training  on  the  standards,  and  enforces  accessibility  standards  for  federally 
funded  facilities”  (U.S.  General  Services  Administration,  2005).  The  Access  Board  fully 
defined  web  accessibility  compliance  guidelines  in  Electronic  and  Information 
Technology  Accessibility  Standards  (Code  of  Federal  Regulations,  2000).  Paragraph 
1 194.22  of  this  regulation  outlines  the  specific  provisions  that  federal  agencies  must 
adhere  to  when  producing  web  pages. 
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Many  of  the  standards  mandated  by  the  Access  Board  provide  for  interoperability 
with  assistive  technologies  and  focus  primarily  on  addressing  visual  disabilities. 

Furthermore,  most  of  the  provisions  are  based  on  checklist  items  from  the  W3C  Web 

•  •••••  •••  ••  •  ^ 
Accessibility  Initiative  (WAI)  Web  Content  Accessibility  Guidelines  version  1.0 

(WCAG  1.0)  which  was  designed  to  help  web  site  designers  meet  the  accessibility  needs 

of  the  Internet  community  (World  Wide  Web  Consortium,  1999a).  WAI  guidelines  are 

divided  into  three  categories: 

Priority  1  checkpoints  that  must  be  satisfied  to  allow  access  to  all  disabled 
groups. 

Priority  2  checkpoints  that  should  be  met  to  remove  significant  access 
barriers. 

Priority  3  checkpoints  that  may  be  addressed  to  improve  access  to  documents. 
(World  Wide  Web  Consortium,  1999b) 

Section  508  web  compliance  items  mandated  by  the  Access  Board  are 

predominately  based  on  Priority  1  and  2  items.  Table  3  summarizes  the  sixteen  Access 

Board  requirements  for  federal  agencies  and  describes  the  relationship  between  each 

Section  508  compliance  item  and  its  corresponding  WCAG  1.0  checkpoints  as  interpreted 

by  Thatcher  (Thatcher,  2005). 

Relationship  Between  Usability  and  Accessibility 

With  an  increasing  emphasis  on  both  accessibility  and  usability,  it  is  reasonable  to 
attempt  to  understand  what,  if  any,  relationship  exists  between  them.  Three  views  are  that 
accessibility  is  a  subset  of  usability,  accessibility  is  a  necessary  but  not  a  sufficient 


Web  Content  Accessibility  Guidelines  2.0  were  available  for  public  comment  at  the  time  of  this  research. 
However  these  updated  guidelines  but  have  not  been  ratified  and  are  not  considered  in  this  thesis  effort. 


29 


Table  3  Section  508  Web  Compliance  Items  (Thatcher,  2005) 


1194.  22 
Paragraph 

Compliance  Item 

Related  WCAG 
1.0  Checkpoint 
and  Priority 

Relationship 

a 

A  text  equivalent  for  every  non-text  element  shall  be 
provided  (e.g.,  via  "alt",  "longdesc",  or  in  element 
content). 

1.1  (PI) 

Similar 

b 

Equivalent  alternatives  for  any  multimedia  presentation 
shall  be  synchronized  with  the  presentation. 

1.4  (PI) 

Same 

c 

Web  pages  shall  be  designed  so  that  all  information 
conveyed  with  color  is  also  available  without  color,  for 
example  from  context  or  markup. 

2.1  (PI) 

Same 

d 

Documents  shall  be  organized  so  they  are  readable 
without  requiring  an  associated  style  sheet. 

6.1  (PI) 

Same 

e 

Redundant  text  links  shall  be  provided  for  each  active 
region  of  a  server-side  image  map. 

1.2  (PI) 

Same 

f 

Client-side  image  maps  shall  be  provided  instead  of 
server-side  image  maps  except  where  the  regions  cannot 
be  defined  with  an  available  geometric  shape. 

9.1  (PI) 

Same 

g 

Row  and  column  headers  shall  be  identified  for  data 
tables. 

5.1  (PI) 

Same 

h 

Markup  shall  be  used  to  associate  data  cells  and  header 
cells  for  data  tables  that  have  two  or  more  logical  levels 
of  row  or  column  headers. 

5.2  (PI) 

Same 

i 

Frames  shall  be  titled  with  text  that  facilitates  frame 
identification  and  navigation. 

12.1  (PI) 

Same 

j 

Pages  shall  be  designed  to  avoid  causing  the  screen  to 
flicker  with  a  frequency  greater  than  2  Hz  and  lower 
than  55  Hz. 

7.1  (PI) 

Same 

k 

A  text-only  page,  with  equivalent  information  or 
functionality,  shall  be  provided  to  make  a  web  site 
comply  with  the  provisions  of  this  part,  when 
compliance  cannot  be  accomplished  in  any  other  way. 

The  content  of  the  text-only  page  shall  be  updated 
whenever  the  primary  page  changes. 

11.4  (PI) 

Same 

1 

When  pages  utilize  scripting  languages  to  display 
content,  or  to  create  interface  elements,  the  information 
provided  by  the  script  shall  be  identified  with  functional 
text  that  can  be  read  by  assistive  technology. 

6.3  (PI) 

6.4  (P2) 

8.1  (P2) 

9.3  (P2) 

508  Less 
Restrictive 

m 

When  a  web  page  requires  that  an  applet,  plug-in  or 
other  application  be  present  on  the  client  system  to 
interpret  page  content,  the  page  must  provide  a  link  to  a 
plug-in  or  applet  that  complies  with  §1 194.21(a)  through 
(1). 

6.3  (PI) 

6.4  (P2) 

8.1  (P2) 

Similar 

n 

When  electronic  forms  are  designed  to  be  completed  on¬ 
line,  the  form  shall  allow  people  using  assistive 
technology  to  access  the  information,  field  elements,  and 
functionality  required  for  completion  and  submission  of 
the  form,  including  all  directions  and  cues. 

9.3  (P2) 

10.2  (P2) 

12.4  (P2) 

Similar 

0 

A  method  shall  be  provided  that  permits  users  to  skip 
repetitive  navigation  links. 

13.5  (P3) 

1.6  (P3) 

508  More 
Specific 

p 

When  a  timed  response  is  required,  the  user  shall  be 
alerted  and  given  sufficient  time  to  indicate  more  time  is 
required. 

None 

Not  in 
WCAG 
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condition  for  usability,  and  that  usability  and  accessibility  are  non-related,  sometimes 
opposing  concepts  (Alexander,  2004). 

Proponents  of  the  first  point  of  view  suggest  that  accessibility  simply  expands 
usability  to  a  larger  population.  Those  who  believe  that  accessibility  is  a  necessary  but 
not  a  sufficient  condition  for  usability  argue  that  accessibility  is  required  for  users  to 
access  data  in  the  first  place  but  that  this  technical  accessibility  alone  is  not  sufficient  to 
produce  a  usable  product.  Others,  primarily  practitioners,  argue  that  strict  technical 
requirements  for  accessibility  often  conflict  with  more  general  usability  principles 
(Alexander,  2004). 

Ultimately,  no  definitive  explanation  has  been  agreed  upon  to  describe  the 
relationship  between  usability  and  accessibility.  However,  common  threads  in  the 
literature  suggest  that  a  relationship  exists,  accessibility  benefits  all  users,  and  usable  sites 
offer  an  improved  experience  to  all  users  (Alexander,  2004;  Nielsen,  2000; 

Vanderheiden,  2005). 

The  Role  of  Guidance  In  Designing  Usable  Web  Sites 

Since  usability  and  accessibility  are  widespread  and  potentially  damaging 
problems  for  web-based  interfaces,  it  is  reasonable  to  ask  why  and  to  explore  what  can  be 
done  to  address  this  issue.  One  of  the  most  likely  reasons  for  the  limited  usability  of 
many  web-based  interfaces  is  that  they  are  being  developed  by  designers  with  limited 
usability  or  user  interface  design  knowledge  (Borges  et  ah,  1996).  However  guidelines 
provide  a  method  to  address  this  knowledge  shortfall.  HCI  guidelines  are  a  mechanism 
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for  transferring  human  factors  knowledge  to  designers  and  can  play  a  valuable  role  as 
references  during  interface  design  (de  Souza  and  Bevan,  1990). 

Unfortunately,  guidelines  used  in  web  page  design  are  typically  not  on  equal 
footing  with  established  HCI  guides.  Ratner  reviewed  a  set  of  common  WWW 
guidelines  and  found  that  of  270  HTML-relevant  recommendations  identified  in  HCI 
guidelines,  only  53  appeared  within  the  WWW  guidelines  (Ratner  et  ah,  1996).  One 
explanation  for  the  small  overlap  observed  is  that  HTML  styled  guides  have  simply 
developed  with  “less  rigor  and  little  reference  to  established  HCI  style  guides  and 
principles”  (Ratner  et  ah,  1996). 

When  valid  HCI  guidelines  are  available  however,  they  can  be  a  valuable  source 
for  improving  web  sites.  In  one  experiment  by  Borges,  three  webpage  designers  were 
provided  a  list  of  17  HCI  guidelines  with  no  additional  explanation  and  asked  to  redesign 
an  existing  home  page.  No  constraints  were  given  other  than  adherence  to  these 
guidelines.  Usability  testing  of  the  redesigned  home  pages  found  an  80  percent 
improvement  in  task  efficiency.  Borges’s  work  demonstrates  that  use  of  web  design 
guidance  based  on  established  HCI  principles  can  generate  measurable  improvements 
(Borges  et  al.,  1998). 

Previous  Air  Force  Intranet  Web  Site  Research 

Previous  AFIT  students  found  that  systems  evaluated  in  their  research  efforts 
suffered  from  low  usability  scores  when  evaluated  against  common  practice  usability 
heuristics.  More  specifically,  the  Air  Force  Knowledge  Now  (AFKN)  web  site  and  the 
web-based  Automated  Civil  Engineer  System  Personnel  Readiness  (ACES-PR  STP) 
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module  of  the  Global  Combat  Support  System  (GCCS)  scored  poorly  using  Peirotti’s 
usability  checklist  (Felax,  2005;  Kastenholz,  2005).  Furthermore,  the  Air  Force 
Knowledge  Now  web  site,  which  was  also  evaluated  for  accessibility,  was  found  to  have 
a  43  percent  compliance  rate  for  Priority  1  WAI  checkpoints  and  47  percent  for  Priority  2 
checkpoints  (Felax,  2005).  One  interesting  finding  by  Kastenholz  was  that  despite  its 
relatively  low  heuristic  compliance  score,  ACE-PR  STP  did  comply  with  applicable  Air 
Force  regulations  and  guidance.  He  also  found  that  usability  scores  were  high  for 
heuristics  addressed  by  Air  Force  guidance  and  low  for  those  absent  from  Air  Force 
guidance.  Overall,  this  may  indicate  that  low  usability  is  due  not  to  non-compliance  with 
Air  Force  standards,  but  to  shortfalls  in  those  standards  (Kastenholz,  2005). 

Current  Research 

The  purpose  of  this  research  effort  is  to  investigate  whether  the  findings  by  Felax 
and  Kastenholz  are  representative  of  other  Air  Force  web  sites  and  to  explore  the 
adequacy  of  the  applicable  Air  Force  and  Department  of  Defense  guidance  for  web  sites 
and  web-based  applications.  Based  on  the  preceding  literature  review,  it  is  hypothesized 
that  some,  but  not  all  usability  and  accessibility  principles  are  contained  in  military 
guidance  and  that  some  overlap  exists  between  usability  and  accessibility  principles.  It  is 
further  hypothesized  that  a  web  site  may  comply  with  some,  all,  or  none  of  these 
standards.  However,  it  is  assumed  that  the  nature  of  the  military  bureaucracy  makes  it 
highly  unlikely  that  Air  Force  intranet  web  sites  will  exhibit  significant  non-compliance 
with  military  guidance.  These  relationships  are  captured  in  the  research  model  in 
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Figure  3  which  depicts  overlap  between  the  various  sources  of  principles  and  guidance 
and  shows  hypothesized  findings  of  web  evaluations. 


The  following  investigative  questions  originally  introduced  in  Chapter  I  are  based 
on  the  findings  of  the  literature  review  and  this  research  model. 

IQ  1 .  How  well  do  Air  Force  intranet  web  sites  follow  common  practice 
usability  principles  identified  in  the  literature? 

IQ2.  How  well  do  Air  Force  intranet  web  sites  adhere  to  the  accessibility 
requirements  of  Section  508  of  the  Rehabilitation  Act? 

IQ3.  Does  a  relationship  exist  between  usability  and  Section  508  accessibility 
compliance  scores? 

IQ4.  Are  common  practice  usability  principles  and  government  mandated 
Section  508  accessibility  requirements  reflected  in  or  referenced  by 
military  guidance  that  is  relevant  to  Air  Force  intranet  web  sites? 

IQ5.  Does  a  relationship  exist  between  compliance  scores  and  the  existence  of 
common  practice  usability  principles  and  Section  508  accessibility 
guidelines  in  military  guidance? 
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Chapter  Summary 

This  chapter  provided  a  review  of  the  relevant  literature.  E-business  and  intranet 
concepts  were  discussed  and  described  in  context  of  the  Air  Force  environment  and 
ongoing  DoD  transformation  efforts.  TAM  was  established  as  a  viable  predictor  of 
intended  information  system  use  and  perceived  usefulness  and  perceived  ease  of  use  were 
established  as  key  detenninants  of  intended  adoption.  Usability  was  defined  in  terms  of 
ease  of  use  and  various  methods  of  evaluating  usability  were  discussed.  Accessibility 
and  Section  508  guidelines  were  described  and  evaluation  methods  discussed.  Past 
research  of  Air  Force  web  sites  that  found  low  usability  and  accessibility  were  cited  as 
justification  for  the  current  research.  A  research  model  was  then  developed  to  visually 
depict  this  study’s  underlying  hypotheses. 
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III.  Methodology 


Chapter  Overview 

This  chapter  presents  the  methodology  used  to  pursue  the  investigative  questions 
outlined  in  Chapters  I  and  II  of  this  study.  First,  the  rationale  for  choosing  a  research 
design  is  presented.  Next,  specific  research  design  issues  are  addressed  and  the  research 
cases  identified.  Data  collection  is  then  discussed  and  the  chapter  concludes  with  an 
overview  of  the  how  this  study  exhibits  the  components  of  quality  research. 

Case  Study  Rationale 

There  are  a  number  of  recognized  approaches  to  research  including  experiment, 
quasi-experiment,  survey,  and  field  study  designs  (Schwab,  2005).  Another  often  used 
method  is  the  case  study  approach.  While  each  strategy  can  be  used  for  exploratory, 
descriptive,  or  explanatory  research,  certain  strategies  lend  themselves  more  to  certain 
purposes.  Deciding  which  method  to  employee  is  critical  to  the  research  effort.  Yin 
provides  three  conditions  for  determining  which  method  is  most  appropriate  (Yin,  2003). 

Yin’s  first  condition  for  choosing  a  design  is  the  form  of  the  research  question. 
Yin  encourages  discretion  in  development  of  the  research  question  and  describes  it  as  the 
most  important  step  in  the  research  effort.  Research  questions  have  both  form  and 
substance,  each  of  which  provide  important  insight  into  the  most  appropriate  research 
strategy  to  employ.  Research  questions  may  take  the  basic  form  of  who,  what,  where, 
how,  or  why.  Exploratory  what  questions  can  be  addressed  by  any  research  strategy. 
However,  prevalence  what  questions  such  as  how  many  are  best  addressed  by  surveys  or 
archival  analysis.  These  strategies  are  also  appropriate  for  who  and  where  questions, 
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especially  where  the  goal  is  to  describe  frequency  of  occurrence  or  to  be  predictive.  How 
and  why  questions  are  often  best  addressed  with  case  studies,  histories,  or  experiments 
(Yin,  2003). 

Yin’s  second  and  third  conditions  consider  the  extent  of  control  the  researcher  has 
over  behavioral  effects  and  whether  the  focus  is  on  contemporary  or  historical  events. 
Experiments  are  appropriate  when  the  researcher  needs  to  have  control  over  behavioral 
events  while  the  other  designs  are  appropriate  when  the  research  needs  little  or  no 
control.  Understandably,  all  designs  except  histories  are  potentially  appropriate  for  the 
study  of  contemporary  events.  Past  events  lend  themselves  to  history  or  archival  analysis 
strategies  (Yin,  2003).  Table  4  summarizes  the  strategies  most  appropriate  for  each 
combination  of  conditions. 


Table  4.  Conditions  for  Different  Research  Strategies  (Yin,  2003) 


Strategy 

Form  of  Research 
Question 

Requires  Control  of 
Behavioral  Events? 

Focuses  on 

Contemporary  Events? 

Archival  Analysis 

who,  what,  where,  how 
many,  how  much 

No 

Yes  /  No 

Case  Study 

how,  why 

No 

Yes 

Experiment 

how,  why 

Yes 

Yes 

History 

how,  why 

No 

No 

Survey 

who,  what,  where,  how 
many,  how  much 

No 

Yes 

Following  Yin’s  reasoning,  a  case  study  is  the  most  appropriate  strategy  for  the 
current  research  effort.  The  purpose  of  this  research  is  to  explore  how  well  Air  Force 
intranet  web  sites  adhere  to  common  practice  guidelines  and  principles  and  how  well 
those  guidelines  and  principles  are  captured  in  military  guidance.  Furthermore,  this 
research  addresses  the  current  state  of  affairs  of  Air  Force  web  sites.  Because  the  goal  of 
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this  research  is  to  understand  the  “as-is”  condition,  the  researcher  neither  requires  nor 
should  assert  any  control  of  behavioral  events. 

Case  Study  Design 

Case  studies  can  include  either  a  single  case  or  multiple  cases.  Furthennore,  the 
purpose  of  research  may  be  to  explore,  describe,  or  explain  (Yin,  2003).  These 
characteristics  create  six  possible  types  of  case  studies  as  shown  in  Table  5. 


Table  5.  Possible  Case  Study  Designs  (Yin,  2003) 


Single  Case  —  Exploratory 

Multiple  Case  —  Exploratory 

Single  Case  --  Descriptive 

Multiple  Case  --  Descriptive 

Single  Case  —  Explanatory 

Multiple  Case  —  Explanatory 

The  current  research  is  conducted  as  a  multiple  case,  explanatory  study.  The  goal 
is  to  explain  the  current  status  of  usability  and  accessibility  of  Air  Force  intranet  web 
sites.  Multiple  cases  will  provide  the  ability  for  cross-case  comparisons  resulting  in  more 
robust  support  for  findings  and  expanded  generalizability  of  the  results. 

Qualitative  vs.  Quantitative  Analysis 

All  research  can  be  described  as  either  quantitative,  qualitative,  or  mixed  method 
which  is  essentially  a  combination  of  both  (Creswell,  2003).  Quantitative  research 
involves  testing  hypothesized  causal  relationships  between  variables.  In  contrast, 
qualitative  research  assumes  very  little  about  possible  relationships  and  instead  seeks  to 
explore  what  variables  lead  to  a  consequence.  Creswell  describes  quantitative  research  as 
using  primarily  “postpositivist  claims  for  developing  knowledge”  and  qualitative  research 
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as  making  “knowledge  claims  based  primarily  on  constructivist  perspectives”  (Creswell, 
2003).  Mixed  method  approaches  “base  knowledge  claims  on  pragmatic  grounds”  and 
often  involve  sequential  collection  of  both  quantitative  and  qualitative  data,  at  the 
researcher’s  discretion,  to  best  understand  the  phenomenon  (Creswell,  2003). 

Yin  contends  that  specific  research  designs  are  not  inherently  more  appropriate 
for  quantitative  or  qualitative  analysis.  Furthermore,  he  argues  that  neither  quantitative 
nor  qualitative  analysis  is  stronger  than  the  other  (Yin,  2003).  Indeed,  Creswell 
acknowledges  that  the  nature  of  modem  research  “is  less  quantitative  versus  qualitative 
and  more  how  research  practices  lie  somewhere  on  a  continuum  between  the  two” 
(Creswell,  2003).  These  arguments  leave  it  to  the  researcher  to  determine  the  method  of 
analysis  that  is  most  appropriate  to  the  situation  studied. 

A  mixed  method  approach  is  taken  in  the  current  study.  The  literature  supports 
Nielsen’s  heuristics  as  one  of  the  most  widely  used  and  cost  effective  methods  of 
usability  inspection.  A  review  of  accessibility  standards  and  Section  508  guidance 
revealed  a  list  of  mandatory  compliance  items  created  by  the  Access  Board.  Evaluations 
based  on  these  heuristics  and  guidelines  are  inherently  qualitative  as  each  inspection  item 
relies  on  the  evaluator’s  subjective  interpretation  (Felax,  2005).  However,  the  resulting 
numeric  compliance  scores  share  some  similarities  with  quantitative  data  and  can  be 
investigated  using  quantitative  methods.  Furthermore,  because  the  current  research 
follows  past  AFIT  research  efforts  on  the  same  subject,  using  similar  methods  as 
employed  in  those  studies  has  the  added  benefit  of  offering  findings  that  are  more  directly 
comparable  with  the  findings  of  those  studies. 
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The  Research  Design 

Yin  proposes  five  primary  components  of  research  design  that  are  especially 
significant  for  case  study  research.  These  components  are  the  research  question, 
propositions,  unit  of  analysis,  logic  linking  the  data  to  the  propositions,  and  criteria  for 
interpreting  the  study’s  findings  (Yin,  2003).  The  following  sections  discuss  these 
components  and  describe  how  the  current  research  incorporates  each. 

The  Research  Question. 

The  first  component  is  the  research  question.  The  research  question  and 
investigative  questions  for  this  research  were  introduced  in  Chapters  I  and  II,  but  merit 
additional  discussion  in  here.  The  research  question  seeks  to  explain  how  well  Air  Force 
intranet  web  sites  follow  common  practice  web  usability  and  Section  508  accessibility 
guidelines.  The  literature  review  detailed  in  Chapter  II  described  the  importance  of 
perceived  ease  of  use  (PEOU)  in  user  acceptance  and  identified  key  components  of 
usability  and  accessibility  that  affect  the  user’s  PEOU.  This  serves  as  the  foundation  for 
IQ1  and  IQ2.  The  literature  review  also  established  theoretical  links  between  usability 
and  accessibility,  supporting  the  need  for  IQ3  which  attempts  to  establish  support  for  a 
relationship  between  usability  and  accessibility.  The  literature  also  validates  the  logical 
role  of  guidance  in  developing  usable  and  accessible  interfaces.  This  leads  to  the  need 
for  IQ  4  and  IQ 5  to  assess  the  adequacy  of  DoD  and  AF  intranet  web  site  guidance. 

Propositions. 

According  to  Yin,  study  propositions  are  necessary  to  focus  the  research  to 
specific  areas  of  interest  within  the  scope  of  the  research.  Case  study  research  can 
become  unwieldy  if  the  researcher  does  not  properly  focus  on  specific  areas.  While  the 
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form  of  the  research  question  may  lead  to  the  decision  to  use  a  case  study,  it  does  not 
necessarily  point  out  exactly  what  should  be  studied.  Yin  states  that  “Only  if  you  are 
forced  to  state  some  propositions  will  you  move  in  the  right  direction”  (Yin,  2003). 

The  first  proposition  of  this  research  is  that  it  will  show  how  well  Air  Force 
intranet  web  sites  fair  when  evaluated  against  common  practice  usability  and  mandatory 
accessibility  criteria.  A  wide  variety  of  usability  and  accessibility  guidelines  have  been 
developed  by  practitioners  and  academia.  This  first  proposition  focuses  the  research 
effort  by  limiting  the  scope  of  usability  and  accessibility  to  the  subset  of  guidelines 
identified  in  the  literature  as  most  relevant.  Specifically,  this  study  focuses  on  Nielsen’s 
heuristics  which  are  the  most  commonly  applied  usability  principles  and  Section  508 
guidelines  for  which  compliance  is  compulsory  for  Federal  agencies. 

The  second  proposition  is  that  the  current  research  use  descriptive  statistics  to 
support  or  fail  to  support  the  theoretical  existence  of  a  relationship  between  usability  and 
accessibility.  Although  several  theories  exist  regarding  the  relationship  between  usability 
and  accessibility,  limited  empirical  evidence  exists  to  support  a  specific  relationship.  The 
second  proposition  provides  a  technique  to  analyze  the  possible  existence  of  the 
relationship,  addressing  IQ3  using  data  obtained  for  IQ1  and  IQ2.  It  is  hoped  that  this 
will  add  to  the  body  of  knowledge  about  the  relationship  between  usability  and 
accessibility  without  expanding  the  scope  of  the  current  study. 

The  third  proposition  is  that  this  research  will  support  or  fail  to  support  that  DoD 
and  Air  Force  guidance  adequately  captures  common  usability  and  mandatory 
accessibility  guidelines.  The  literature  supports  the  role  of  guidance  in  improving 
usability  and  accessibility  by  providing  access  to  basic  design  principles.  However,  this 
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guidance  could  come  from  a  variety  of  sources  and  evaluating  all  possible  sources  is 
beyond  the  resources  available  for  the  current  research.  Therefore,  the  third  proposition 
narrows  the  research  to  focus  solely  on  military  guidance  documents,  the  use  of  which 
may  be  correlated  with  differences  in  compliance  scores  between  different  aspects  of 
usability. 

Unit  of  Analysis. 

The  unit  of  analysis  is  closely  related  to  the  definition  of  the  case  itself. 
According  to  Yin,  the  unit  of  analysis  should  be  driven  by  the  research  question  and 
address  the  entity  that  is  the  focal  point  of  the  research.  It  is  also  important  that  the  unit 
of  analysis  remain  distinguishable  from  the  context  of  the  study.  For  comparability  to 
other  studies,  definition  of  the  unit  of  analysis  should  be  similar  to  the  definition  used  in 
those  studies.  Once  chosen,  other  clarifications  must  be  made  to  draw  clear  boundaries 
between  the  unit  of  analysis  and  related  units  and  to  detennine  the  appropriate  timeframe 
for  analysis  (Yin,  2003). 

This  research  looks  at  six  Air  Force  web  sites  used  to  facilitate  transactions  or 
information  exchange  between  a  host  organization  and  an  authorized  user  for  a  business 
purpose  other  than  those  with  the  general  public.  The  unit  of  analysis  is  the  individual 
Air  Force  intranet  web  site.  The  timeframe  in  question  is  limited  to  the  time  of  data 
collection.  Data  collection  began  in  July  2005  and  concluded  in  January  2006.  All  web 
site  evaluations  were  performed  within  the  time  period  and  only  guidance  existing  prior 
to  the  evaluation  is  considered  by  the  analysis.  This  definition  of  the  unit  of  analysis  is 
consistent  with  that  used  in  past  research  efforts  (Felax,  2005;  Kastenholz,  2005). 
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Logic  Linking  the  Data  to  the  Propositions. 

Once  the  unit  of  analysis  is  defined,  the  next  step  is  to  address  how  the  data  will 
be  used  in  the  context  of  the  propositions.  Two  common  approaches  are  pattern 
matching  and  cross  case  comparisons.  Pattern  matching  compares  the  research  data  with 
a  predicted  pattern  or  alternative  patterns.  Cross  case  comparisons  consider  how  the  data 
compare  across  the  units  of  analysis  from  each  case  study. 

This  research  study  employees  both  techniques.  Cross  case  comparison  is  used  to 
evaluate  how  well  Air  Force  web  sites  adhere  to  usability  and  accessibility  standards. 
Data  for  each  web  site  are  compared  to  identify  patterns  in  compliance.  Pattern  matching 
is  then  used  to  determine  if,  as  theory  predicts,  web  site  usability  and  accessibility  scores 
are  affected  by  the  quality  of  guidance.  The  quality  of  guidance  is  measured  by  the 
amount  of  overlap  between  military  guidance  and  common  practice  usability  and 
accessibility  principles. 

Criteria  for  Interpreting  the  Data. 

In  qualitative  research,  the  criteria  for  what  constitutes  a  significant  effect  is  not 
as  clear  as  in  quantitative  research.  Therefore  it  can  be  difficult  to  ascertain  specific 
cutoffs  or  to  determine  how  well  patterns  must  match  to  be  considered  a  match.  Yin 
cautions  the  researcher  against  establishing  constraints  for  pattern  matching  that  are 
unnecessarily  restrictive.  Sometimes  patterns  need  only  be  visible  to  the  eye,  while  other 
situations  may  call  for  closer  scrutiny  depending  on  the  theoretical  underpinnings  of  the 
research  (Yin,  2003). 
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Criteria  for  Investigative  Question  1. 

Pierotti  does  not  provide  explicit  guidance  for  analyzing  the  results  of  her 
usability  heuristic  checklist,  located  at  Appendix  A,  which  is  used  to  answer  IQ  1 .  Rather, 
she  encourages  the  researcher  to  use  reasonable  judgment  in  assigning  a  point  scale  to 
each  item  according  to  the  nature  of  the  project  undertaken  (Pierotti,  1996).  To  build  on 
the  existing  body  of  research  on  Air  Force  web  sites,  this  research  follows  the  same 
scoring  methodology  used  by  Felax  and  Kastenholz  (Felax,  2005;  Kastenholz,  2005) 
which  is  based  on  concepts  for  evaluating  homepage  design  outlined  by  Nielsen  and 
Tahir  (Nielsen  and  Tahir,  2002).  Checklist  items  receive  a  score  of  “1”  for  applicable 
criteria  that  are  met,  “0”  for  applicable  criteria  that  are  not  met,  and  no  score  for  non- 
applicable  criteria.  Compliance  for  each  heuristic  is  calculated  as  a  percentage  of 
conformance  with  applicable  items.  The  overall  compliance  rating  for  each  site  is  then 
calculated  by  giving  equal  weight  to  each  section  and  summing  the  resulting  scores,  with 
a  maximum  possible  score  of  100  percent.  The  overall  usability  scores  are  evaluated 
using  Nielsen  and  Tahir’s  grading  criteria,  summarized  in  Table  6,  which  requires  a  mean 
usability  score  greater  than  80  percent  to  be  considered  usable  (Nielsen  and  Tahir,  2002). 

Criteria  for  Investigative  Question  2. 

The  criteria  for  evaluating  IQ2  is  similar  to  that  used  for  IQ  1 .  Cases  are  evaluated 
against  the  sixteen  Section  508  mandatory  compliance  items  established  by  the  Access 
Board  and  identified  in  the  literature  review.  Checklist  items  receive  a  score  of  “1”  for 
applicable  criteria  that  are  met,  “0”  for  applicable  criteria  that  are  not  met,  and  no  score 
for  non-applicable  criteria.  The  mean  accessibility  score  for  each  web  site  is  calculated 
as  a  percentage  of  applicable  items  meet.  Based  on  the  mandatory  nature  of  Section  508 
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Table  6.  Web  Site  Usability/Accessibility  Compliance  Rate  (Felax,  2005;  Nielsen  and  Tahir,  2002) 


Compliance 

Rate 

Description 

90  to  95% 

Perfect  web  site.  Follows  almost  all  usability/accessibility  guidelines  that  apply  to  that 
particular  site  and  does  something  different,  but  appropriate,  in  the  remaining  cases. 

80  to  90% 

Web  site  is  in  good  shape.  Consider  making  a  few  minor  fixes  to  areas  where  the  site 
violated  guidelines. 

50  to  80% 

Start  a  redesign  project  to  produce  a  new  homepage.  Your  current  homepage  is 
definitely  not  a  disaster,  but  it  is  bad  enough  that  isolated  modifications  to  individual 
areas  will  not  suffice. 

below  50% 

The  site  is  probably  not  serving  your  customers  well  with  the  current  approach  to  web 
design.  Most  likely,  you  should  abandon  the  entire  current  site  and  start  over  from 
scratch.  Rethink  your  Internet  strategy  and  base  your  new  approach  on  studies  of  your 
customers  and  their  real  needs. 

compliance  for  government  agencies,  it  seems  intuitive  to  establish  criteria  that  web  sites 
must  score  100%  to  be  considered  fully  compliant.  However,  opinions  of  what  specific 
criteria  must  be  meet  to  satisfy  each  guideline  may  vary  based  on  the  context  of  the  study 
and  the  background  of  the  evaluator.  Therefore  use  of  a  single  evaluator  dictates  that 
some  flexibility  be  used  in  interpreting  the  results.  For  the  purposes  of  this  study, 

Nielsen  and  Tahir’s  grading  scale  is  applied  to  accessibility  scores.  An  accessibility 
score  of  greater  than  or  equal  to  80  percent  is  considered  accessible  just  as  a  usability 
score  of  greater  than  or  equal  to  80  percent  is  considered  usable.  This  accessibility 
criteria  is  consistent  with  that  used  by  Felax  (Felax,  2005)  and  facilitates  comparison 
between  this  study  and  his. 

Criteria  for  Investigative  Question  3. 

Scores  obtained  from  IQ1  and  IQ2  are  used  to  answer  IQ3.  Usability  and 
accessibility  scores  are  evaluated  using  descriptive  statistics  to  identify  possible 
correlation  between  the  two.  Due  to  the  small  number  of  samples  used  in  this  study, 
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robust  linear  regression  cannot  be  used  to  approximate  the  strength  of  any  correlation. 
However,  descriptive  methods  such  as  comparing  the  mean  usability  and  accessibility 
scores  for  each  web  site  will  provide  sufficient  evidence  to  indicate  whether  a  strong 
correlation  exists  between  usability  and  accessibility  compliance  scores. 

Criteria  for  Investigative  Question  4. 

Military  guidance  documents  are  reviewed  to  determine  whether  they  are 
applicable  to  Air  Force  intranet  web  site  design,  maintenance,  and/or  use  and  to  establish 
whether  compliance  with  that  guidance  is  mandatory.  Next,  each  relevant  document  is 
evaluated  to  determine  which,  if  any,  of  Nielsen’s  heuristics  it  contains  and  whether  it 
specifies  Section  508  accessibility  compliance.  The  number  of  Nielsen’s  heuristics  that 
are  found  in  military  each  military  guidance  document  is  reported  in  terms  of  the  raw 
number  of  heuristics  found  and  as  a  percentage  of  heuristics  found  in  military  guidance. 
Military  guidance  is  given  credit  for  including  a  specific  heuristic  if  there  is  evidence  of 
that  heuristic  in  any  of  the  mandatory  guidance  documents.  The  number  of  heuristics 
represented  at  least  once  in  military  guidance  is  divided  by  the  total  number  of  heuristics 
to  establish  the  percentage  of  heuristics  captured. 

Higher  percentages  of  overlap  demonstrate  higher  potential  for  transferring 
knowledge  of  established  design  principles  while  lower  percentages  of  overlap  create 
fewer  opportunities  for  knowledge  transfer.  Following  Yin’s  guidance  against  being 
overly  confined  by  artificial  boundaries  (Yin,  2003),  this  research  establishes  a  threshold 
criteria  that  at  least  50  percent  of  Nielsen’s  heuristics  should  be  found  in  military  policy. 
Existence  of  this  simple  majority  of  Nielsen’s  heuristics  in  military  policy  is  sufficient  to 
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establish  that  basic  common  practice  guidelines  are  at  least  considered  by  military 
guidance. 

Criteria  for  Investigative  Question  5. 

To  evaluate  IQ5,  compliance  scores  obtained  from  IQ1  and  IQ2  are  compared 
against  the  findings  of  IQ  4.  Descriptive  methods  are  used  to  assess  whether  there  is  a 
correlation  between  the  existence  of  common  practice  guidelines  in  military  guidance  and 
the  compliance  score  for  each  heuristic  and  Section  508  compliance  item.  Due  to  the 
small  number  of  samples  used  in  this  study,  robust  linear  regression  cannot  be  used  to 
approximate  the  strength  of  any  correlation.  However,  descriptive  methods  such  as 
comparing  the  mean  compliance  score  for  each  heuristic  with  the  number  of  times  a 
heuristic  is  mentioned  in  military  guidance  will  provide  sufficient  evidence  to  indicate 
whether  a  strong  correlation  exists.  Table  7  summarizes  the  criteria  for  interpreting  the 
data  for  each  investigative  question. 


Table  7.  Summary  of  Criteria  for  Interpreting  the  Data 


Investigative 

Question 

Criteria  for  Interpreting  the  Data 

IQ1 

Usability  compliance  scores  are  evaluated  using  Nielsen  and  Tahir’s 
homepage  scoring  guidelines.  Sites  that  have  a  mean  usability  score 
greater  than  80%  are  considered  usable. 

IQ2 

Accessibility  compliance  scores  are  evaluated  using  Nielsen  and  Tahir’s 
homepage  scoring  guidelines.  Sites  that  have  a  mean  accessibility  greater 
than  80%  are  considered  accessible. 

IQ  3 

Usability  and  accessibility  compliance  scores  appear  to  be  correlated. 

IQ4 

Percent  of  total  common  practice  principles  that  are  found  in  military 
guidance.  At  least  50%  of  Nielsen’s  heuristics  should  be  found  in  the 
military  policy  to  establish  that  common  practice  guidelines  are  at  least 
considered  by  military  guidance. 

IQ5 

Existence  of  common  practice  principles  in  military  policy  appears  to  be 
correlated  with  respective  compliance  scores. 
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Case  Selection 


In  a  multiple  case  study  approach,  the  motivation  behind  case  selection  is 
replication  of  results  rather  than  traditional  statistics  sampling  logic.  Therefore,  case 
selection  must  be  carefully  conducted  and  should  consider  prior  knowledge  of  expected 
outcomes.  Cases  selected  for  literal  replication  should  produce  similar  results.  However, 
cases  may  be  selected  that  produce  contrasting  results  but  for  predictable  reasons  to 
establish  theoretical  replication.  In  a  best  case  scenario,  a  mix  of  both  types  of  cases  is 
used  to  provide  both  literal  and  theoretical  replication  (Yin,  2003). 

The  goal  of  replication  also  impacts  the  number  of  cases  that  must  be  used  in  a 
study  as  well  as  which  cases  are  used.  Because  sampling  logic  is  not  used  too  capture  an 
entire  population  or  a  representative  sample,  traditional  methods  to  detennine  the  optimal 
number  of  cases  is  not  applicable.  Yin  states  that  the  number  of  cases  to  be  used  is  at  the 
researcher’s  discretion  and  depends  primarily  on  the  certainty  of  the  results  required.  A 
larger  number  of  cases  results  in  greater  certainty.  Slight  nuances  in  theory  require  more 
cases  to  detect  as  do  a  larger  number  of  external  variables  that  affect  the  outcome  of  the 
case  (Yin,  2003). 

Other  criteria  for  case  selection  include  how  representative  or  unique  a  case  is,  a 
special  arrangement  with  the  case,  or  access  to  the  case.  Cases  that  are  especially 
representative  of  other  cases  or  unique  from  other  cases  are  well  suited  for  single  case 
designs.  Often  a  case  is  selected  because  the  researcher  can  gain  access  to  the  necessary 
resources  to  conduct  research.  Regardless  of  the  rationale  for  case  selection,  cases  should 
be  selected  prior  to  fonnal  data  collection  (Yin,  2003). 
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Cases  for  this  research  were  selected  based  the  purpose  of  the  intranet  site  and  on 
accessibility  to  the  case.  The  literature  review  established  that  intranet  web  sites  are  used 
for  a  variety  of  purposes.  Within  the  military,  two  basic  types  of  intranet  sites  are  those 
that  facilitate  a  transaction  and  those  that  facilitate  infonnation  exchange.  Specific 
purposes  of  Air  Force  web  sites  include  conducting  administrative,  logistical,  and 
operational  transactions,  and  facilitating  information  exchanges.  A  number  of  potential 
cases  were  randomly  selected  from  the  Air  Force  Portal’s  web  links  page  and  screened  to 
identify  several  cases  from  each  basic  function.  Aside  from  the  site’s  purpose,  primary 
consideration  was  given  to  accessibility  to  the  case.  Therefore  the  web  site  must  exist  on 
an  unclassified  network  and  not  require  special  access  or  be  willing  to  grant  special 
access  to  evaluators. 

Official  points  of  contact  were  reached  at  twenty  potential  cases.  Of  these,  eight 
positive  responses  were  received.  One  was  removed  from  consideration  because  it  was 
identified  as  a  subset  of  a  web  site  evaluated  in  a  previous  research  effort.  A  second  was 
removed  due  to  the  organization’s  limited  ability  to  support  the  research.  The  remaining 
six  web  sites  are  included  in  this  research.  Anonymity  was  promised  as  a  condition  of 
participation.  The  cases  are  described  in  Table  8. 

Data  Collection 

Usability  scores  were  obtained  by  several  evaluators  for  each  web  site. 

Evaluators  were  selected  based  on  participation  in  an  electronic  business  class  taught  at 
AFIT.  All  evaluators  were  Master’s  level  students.  Each  evaluator  received  four  hours 
of  instruction  in  usability  principles  and  heuristic  evaluation  techniques.  A  pilot  study 
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Table  8.  Research  Cases  Used  in  This  Study 


Case 

Description 

A 

Intranet  site  offering  functional  area  guidance,  forms,  and  best  practices 

B 

Intranet  site  offering  functional  area  guidance,  best  practices,  and  information 
exchange 

C 

Interactive  intranet  site  for  transactions  and  updating  information 

D 

Intranet  site  offering  wide-range  a  of  information  and  guidance  for  download 

E 

Intranet  site  offering  interactive,  user  configurable  warfighter  support  information 

F 

Interactive  intranet  site  for  transactions  and  updating  information 

was  then  conducted  using  the  Air  Force  Knowledge  Know  web  site.  Each  evaluator’s 
results  were  compared  to  the  results  of  Felax’s  previous  heuristic  evaluation  of  the  same 
web  site  (Felax,  2005).  Evaluators  then  received  two  hours  of  additional  instruction 
focused  on  weak  areas  identified  during  the  pilot  study. 

Following  training,  each  evaluator  was  assigned  two  intranet  sites.  With  the 
exception  of  the  primary  researcher  who  inspected  all  six  web  sites,  evaluator  to  web  site 
assignments  were  made  randomly  with  the  only  constraint  being  that  no  combination  of 
inspectors  was  repeated  for  any  two  web  sites.  Each  site  was  evaluated  by  at  least  two 
evaluators  and  the  primary  researcher,  resulting  in  a  minimum  of  three  inspectors  per  web 
site.  Each  evaluator  was  given  a  six-week  window  to  complete  his  inspections. 
Completed  inspections  were  held  by  the  thesis  advisor  until  the  primary  researcher 
completed  inspections  of  all  six  web  sites. 

Due  to  the  more  technical  nature  of  the  accessibility  guidelines  over  usability 
heuristics,  accessibility  inspections  were  performed  only  by  the  primary  researcher. 
Accessibility  inspections  were  performed  following  completion  of  usability  inspections. 

A  list  of  applicable  military  guidance  documents  was  developed  by  reviewing  available 
military  guidance.  The  resulting  list  of  relevant  guidance  was  then  reviewed  for  overlap 
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with  Nielsen’s  usability  heuristics  and  the  Access  Board’s  Section  508  compliance 
guidelines  for  web  sites. 

Review  of  military  guidance  was  performed  by  the  primary  evaluator  following 
completion  of  the  usability  and  accessibility  inspections.  Military  policy  documents  were 
pulled  from  publicly  available  information  sources  and  do  not  include  any  limited 
distribution  documents.  The  findings  were  complied  in  a  database  that  is  located  at 
Appendix  B. 

Research  Quality 

To  be  considered  high  quality,  research  efforts  must  maximize  for  conditions  — 
construct  validity,  internal  validity,  external  validity,  and  reliability  (Yin,  2003).  This  is 
as  true  for  case  study  designs  as  for  other  research  methods.  The  following  discussion 
describes  how  the  current  research  effort  addresses  each  of  these  conditions. 

Construct  Validity. 

Construct  validity  refers  to  how  well  the  measured  constructs  accurately  capture 
the  theoretical  construct  it  represents  (Schwab,  2005).  The  literature  review  for  this 
research  validated  the  widespread  consensus  among  experts  that  Nielsen’s  heuristics 
satisfactorily  capture  the  primary  tenants  of  usable  HCI  design.  In  addition,  the  literature 
review  clearly  creates  and  maintains  a  chain  of  evidence  that  supports  the  need  for 
usability  based  on  PEOU  as  a  detenninant  of  technology  acceptance,  the  importance  of 
accessibility  as  mandated  by  Federal  law,  and  the  role  of  intranets  in  the  Air  Force 
environment.  Furthermore,  Borges’s  research  validates  the  role  of  guidance  in 
developing  usable  and  accessible  web  sites. 
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Internal  Validity. 

Internal  validity  occurs  when  scores  on  the  independent  variable  are  correlated 
with  scores  on  the  dependent  variable,  indicating  the  ability  of  the  independent  variable 
to  predict  changes  in  the  dependent  variable  (Schwab,  2005).  Case  study  methodology 
addresses  internal  validity  during  data  analysis.  As  discussed  above  in  Logic  Linking  the 
Data  to  the  Propositions,  the  multi-case  design  of  this  research  provides  the  ability  for 
pattern  matching  and  cross  case  comparisons.  This  creates  two  sources  evidence 
supporting  the  link  between  the  dependent  and  independent  variables. 

External  Validity. 

External  validity  refers  to  the  appropriate  generalization  of  findings  to  cases 
outside  those  studied.  In  case  study  research,  external  validity  is  established  by 
replicating  the  results  in  similar  circumstances  and  environments  where  theory  suggests 
the  same  findings  will  occur  (Yin,  2003).  This  research  effort  applies  well  established 
usability  and  accessibility  theory  to  a  range  of  Air  Force  intranet  web  sites.  The  mix  of 
theoretical  and  literal  replication  used  to  compare  and  contrast  the  cases  in  this  study 
extends  the  generalizability  of  the  findings.  However,  discretion  must  be  used  so  as  not 
to  overreach  the  extent  of  generalization  that  is  appropriate. 

Reliability. 

Reliability  is  a  primary  goal  of  all  high-quality  empirical  research.  Reliability 
refers  to  the  amount  of  variance  in  the  research  and  other  researchers’  ability  to  replicate 
research  findings  (Schwab,  2005;  Yin,  2003).  Two  sources  of  reliability  are  use  of  a  case 
study  protocol  and  a  case  study  database.  This  effort  clearly  outlines  the  protocol  used. 
Furthermore,  all  relevant  data  each  web  site  is  included  in  the  main  chapters,  tables,  and 
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Appendix  B  of  this  effort.  Every  effort  has  been  taken  to  make  this  research  transparent 
for  future  researchers  to  understand,  duplicate,  and  reference  with  confidence. 

Chapter  Summary 

This  chapter  explained  the  rationale  for  using  a  case  study  approach  and  outlined 
the  research  design.  The  unit  of  analysis  was  clearly  defined  as  the  individual  intranet 
web  site  and  the  cases  used  in  this  study  were  introduced.  Criteria  for  interpreting  the 
findings  were  developed  based  on  the  literature  review  summarized  in  Chapter  II.  The 
chapter  concluded  with  a  discussion  of  the  steps  taken  to  ensure  that  the  research  effort  is 
of  high  quality. 
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IV.  Results 


Investigative  Question  1:  Air  Force  Web  Site  Usability  Inspection  Results 

Initial  inspection  of  the  data  revealed  that  evaluator  5  and  the  primary  evaluator 
tended  to  give  higher  usability  compliance  scores  than  other  evaluators  as  shown  in 
Table  9.  It  was  also  noted  that  evaluators  4  and  7  gave  somewhat  lower  usability  scores. 
These  nuances  contribute  to  a  range  of  usability  scores  for  each  web  site.  However  this  is 
not  an  unexpected  outcome  but  rather  is  a  result  of  evaluators  placing  different  emphasis 
in  different  areas.  Indeed,  this  is  the  crux  of  Nielsen’s  discount  usability  method.  His 
research  indicates  that  at  least  three  evaluators  are  needed  for  a  thorough  usability 
inspection  (Nielsen  and  Mack,  1994). 


Table  9.  Usability  Scores  by  Case  and  Evaluator 


Evaluator 

Case  A 

Case  B 

CaseC 

Case  D 

Case  E 

Case  F 

1 

— 

0.56 

— 

— 

0.59 

— 

2 

— 

0.55 

— 

0.57 

— 

— 

3 

0.71 

— 

— 

0.80 

— 

— 

4 

— 

— 

0.36 

— 

0.44 

— 

5 

— 

0.77 

— 

0.83 

— 

— 

6 

— 

— 

0.62 

— 

— 

0.70 

7 

0.58 

0.43 

— 

— 

— 

— 

8 

— 

0.68 

— 

— 

— 

0.68 

9 

— 

— 

0.55 

0.49 

— 

— 

Primary  Evaluator 

0.75 

0.67 

0.69 

0.77 

0.63 

0.68 

Figure  4  visually  depicts  the  mean  usability  scores  given  by  each  evaluator  for 
each  web  site  inspected.  The  graph  shows  that  the  mean  usability  score  for  each  case  is 
centered  within  the  individual  evaluator  scores  for  that  web  site.  This  indicates  that  no 
evaluator’s  data  is  exerting  an  undue  influence  on  the  mean  usability  score  for  any  case. 
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It  is  also  noted  that  the  number  of  evaluators  varies  across  each  cases.  This 
variation  is  due  to  incomplete  and  highly  erratic  data  provided  by  two  of  the  evaluators 
which  prevented  their  inspections  from  being  included  in  this  study.  Removal  of  these 
two  evaluators  resulted  in  an  unbalanced  number  of  evaluators  per  case.  However,  the 
minimum  number  of  three  evaluators  per  case  recommended  by  Nielsen  is  met  for  all 
cases. 


Cross  Case  Comparisons. 

The  following  paragraphs  describe  the  results  of  the  web  site  usability 
inspections.  Results  are  detailed  first  by  heuristic  and  then  summarized  in  a  collective 
discussion  of  the  findings. 
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Visibility  of  System  Status. 


The  first  heuristic,  visibility  of  system  status,  posits  that  the  system  should  keep 
users  informed  about  what  is  happening.  As  shown  in  Table  10,  the  average  score  for 
this  heuristic  is  .68  with  a  high  score  of  .87  for  Case  F  and  a  low  score  of  .50  for  Case  C. 
Failing  to  keep  the  user  informed  of  system  status  when  observable  delays  exist  was  the 
checklist  item  requiring  the  greatest  attention  for  this  heuristic. 


Table  10.  Average  Usability  Scores  for  Each  Case  by  Heuristic 


Heuristic 

Case  A 

Case  B 

CaseC 

Case  D 

Case  E 

Case  F 

Overall 

1  Visibility  of  System  Status 

0.74 

0.63 

0.50 

0.78 

0.56 

0.87 

0.68 

2  Match  Between  System  and  the  Real 
World 

0.78 

0.70 

0.64 

0.75 

0.69 

0.85 

0.73 

3  User  Control  and  Freedom 

0.57 

0.68 

0.60 

0.71 

0.66 

0.61 

0.64 

4  Consistency  and  Standards 

0.84 

0.63 

0.70 

0.79 

0.74 

0.85 

0.76 

5  Help  Users  Recognize,  Diagnose,  and 
Recover  From  Errors 

0.79 

0.78 

0.54 

0.75 

0.44 

0.77 

0.68 

6  Error  Prevention 

0.68 

0.46 

0.41 

0.61 

0.26 

0.72 

0.52 

7  Recognition  Rather  Than  Recall 

0.66 

0.54 

0.46 

0.65 

0.54 

0.65 

0.58 

8  Flexibility  and  Efficiency  of  Use 

0.20 

0.49 

0.41 

0.31 

0.32 

0.50 

0.37 

9  Aesthetic  and  Minimalist  Design 

0.74 

0.79 

0.56 

0.71 

0.69 

0.76 

0.71 

10  Help  and  Documentation 

0.43 

0.24 

0.50 

0.70 

0.17 

0.30 

0.39 

11  Skills 

0.58 

0.63 

0.58 

0.69 

0.55 

0.56 

0.60 

12  Pleasurable  and  Respectful  Interaction 
with  the  User 

0.83 

0.67 

0.72 

0.68 

0.75 

0.61 

0.71 

13  Privacy 

1.00 

0.72 

0.58 

0.92 

0.83 

0.89 

0.82 

Overall 

0.68 

0.61 

0.55 

0.69 

0.55 

0.69 

0.63 

|  |  Indicates  scores  lower  than  .50 

|  |  Indicates  scores  higer  than  .80 

Match  Between  System  and  the  Real  World. 

The  second  heuristic,  match  between  system  and  the  real  world,  suggests  that  the 
system  interface  should  use  conventions  that  are  familiar  to  the  user.  The  average  score 
for  this  heuristic  is  .73  with  a  high  score  of  .85  for  Case  F  and  a  low  score  of  .64  for 
Case  C.  Failing  to  follow  a  natural  sequence  for  menu  choices  was  the  most  duplicated 
violation  across  all  evaluators  and  web  sites. 
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User  Control  and  Freedom. 


The  third  heuristic,  user  control  and  freedom,  recommends  that  users  be  provided 
an  easy  mechanism  to  back  out  of  mistakes  including  undo  and  redo  functionality.  The 
average  score  for  this  heuristic  is  .64  with  a  high  score  of  .7 1  for  Case  D  and  a  low  score 
of  .57  for  Case  A.  The  most  common  problems  involve  failure  to  provide  an  undo 
capability  and  no  ability  for  users  to  set  their  own  system,  session  or  screen  defaults. 

Consistency  and  Standards. 

The  fourth  heuristic,  consistency  and  standards,  advocates  use  of  platform-wide 
conventions  that  prevent  user  confusion  by  providing  consistent  meaning  for  words  and 
actions.  The  average  score  for  this  heuristic  is  .76  with  a  high  score  of  .85  for  Case  F  and 
a  low  score  of  .63  for  Case  B.  Lack  of  consistency  in  the  location  and  format  of  online 
instructions  was  the  most  frequently  identified  problem. 

Help  Users  Recognize,  Diagnose,  and  Recover  From  Errors. 

The  fifth  heuristic,  help  users  recognize,  diagnose,  and  recover  from  errors,  states 
that  errors  messages  should  clearly  state  the  problem  and  suggest  a  constructive  solution. 
The  average  score  for  this  heuristic  is  .68  with  a  high  score  of  .79  for  Case  A  and  a  low 
score  of  .44  for  Case  E.  The  most  commonly  identified  problems  were  error  messages 
that  placed  blame  on  the  user  instead  of  the  system  and  failure  to  place  the  cursor  in  or 
highlight  the  field  in  which  the  error  occurred. 

Error  Prevention. 

The  sixth  heuristic,  error  prevention,  calls  for  the  system  to  prevent  user  errors 
whenever  possible.  The  average  score  for  this  heuristic  is  .52  with  a  high  score  of  .72  for 
Case  F  and  a  low  score  of  .26  for  Case  E.  Failure  to  visually  indicate  the  maximum  field 
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length  or  the  number  of  character  spaces  remaining  in  a  field  was  the  most  common  issue 
identified  by  inspectors. 

Recognition  Rather  Than  Recall. 

The  seventh  heuristic,  recognition  rather  than  recall,  recommends  that  the  system 
should  provide  visible  objects,  actions,  options,  and  instructions  rather  than  requiring  the 
user  to  remember  information  from  screen  to  screen.  The  average  score  for  this  heuristic 
is  .58  with  a  high  score  of  .66  for  Case  A  and  a  low  score  of  .46  for  Case  C.  Failing  to 
make  the  first  word  in  each  menu  choice  the  most  important  word  and  failure  to  provide 
an  online  spatial  menu  map  were  the  most  frequently  identified  problems. 

Flexibility  and  Efficiency  of  Use. 

The  eight  hcuri  stic,  flexibility  and  efficiency  of  use,  states  that  the  system  should 
allow  users  to  tailor  frequent  actions  and  provide  accelerators  for  expert  users.  The 
average  score  for  this  heuristic  is  .37  with  a  high  score  of  .50  for  Case  F  and  a  low  score 
of  .20  for  Case  A.  The  most  commonly  identified  issue  was  no  provision  for  keyboard 
shortcuts. 

Aesthetic  and  Minimalist  Design. 

The  ninth  heuristic,  aesthetic  and  minimalist  design,  recognizes  that  limited 
screen  space  is  available  and  only  the  most  relevant  information  to  the  task  at  hand 
should  be  presented.  The  average  score  for  this  heuristic  is  .71  with  a  high  score  of  .79 
for  Case  B  and  a  low  score  of  .56  for  Case  C.  Displaying  more  information  than  is 
essential  to  decision  making  was  the  most  commonly  identified  problem. 
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Help  and  Documentation. 

The  tenth  heuristic,  help  and  documentation,  recommends  that  help  should  be 
readily  available,  useful,  and  task  focused.  The  average  score  for  this  heuristic  is  .39, 
with  a  high  score  of  .70  for  Case  D  and  a  low  score  of .  17  for  Case  E.  The  most 
frequently  identified  shortcomings  were  lack  of  context-sensitive  help  and  no  ability  to 
change  the  level  of  detail  available. 

Skills. 

The  eleventh  heuristic,  skills,  states  that  the  system  should  augment  and  leverage 
user  skills  rather  than  attempt  to  replace  them.  The  average  score  for  this  heuristic  is  .60 
with  a  high  score  of  .69  for  Case  D  and  a  low  score  of  .55  for  Case  E.  Not  automatically 
focusing  the  cursor  on  the  field  users  are  most  likely  to  need  was  the  most  commonly 
identified  problem. 

Pleasurable  and  Respectful  Interaction  with  the  User. 

The  twelfth  heuristic,  pleasurable  and  respectful  interaction  with  the  user, 
suggests  that  the  user  should  be  treated  respectfully  and  the  interface  should  be 
aesthetically  pleasing  in  order  to  increase  the  user’s  quality  of  work-life.  The  average 
score  for  this  heuristic  is  .71  with  a  high  score  of  .83  for  Case  A  and  a  low  score  of  .61 
for  Case  F.  The  most  frequently  identified  issue  was  not  using  color  to  draw  user 
attention,  communicate  organization  and  relationships,  or  indicate  status  changes. 

Privacy. 

The  thirteenth  heuristic,  privacy,  recognizes  the  importance  of  helping  the  user 
protect  personal  and  private  information.  The  average  score  for  this  heuristic  is  .82  with 
a  high  score  of  1 .00  for  Case  A  and  a  low  score  of  .58  for  Case  C.  Problems  identified  by 
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this  heuristic  appeared  to  be  randomly  distributed  with  no  single  item  accounting  for  a 
disproportion  amount  of  negative  findings. 

Heuristic  Compliance  Scores  Compared. 

Of  all  thirteen  heuristics  inspected,  only  privacy  received  a  mean  score  greater 
than  .80  across  all  web  sites.  Two  heuristics,  flexibility  and  efficiency  of  use  and  help  and 
documentation,  received  mean  scores  worse  than  .50  when  averaged  across  all  sites. 
flexibility  and  efficiency  of  use  scores  were  lower  than  .50  for  all  but  one  site  while  only 
two  sites  scored  .50  or  above  for  help  and  documentation. 

The  small  number  of  six  sample  web  sites  for  each  heuristic,  combined  with  the 
wide  variability  of  evaluator  scores  within  each  heuristic  makes  statistical  comparison 
difficult.  However  the  data  clearly  indicates  that  privacy  is  the  heuristic  best  addressed 
by  the  six  web  sites  and  that  all  sites  performed  poorly  on  the  flexibility  and  efficiency  of 
use  and  help  and  documentation  heuristics.  Table  1 1  shows  that  the  scores  for  the  latter 
are  statistically  different  from  privacy  while  all  other  heuristics  fall  within  a  range 
between  .50  and  .80.  Figure  5  visually  reinforces  these  findings,  clearly  showing  that 
scores  for  the  majority  of  the  heuristics  fall  between  .50  and  .80. 

Based  on  the  established  criteria,  privacy  is  in  good  shape  while  flexibility  and 
efficiency  of  use  and  help  and  documentation  are  not  serving  customers  well. 

Furthermore,  the  remaining  ten  heuristics  which  scored  between  .50  and  .80  are  not  a 
disaster,  but  are  bad  enough  that  isolated  modifications  to  individual  areas  will  not  suffice 
(Felax,  2005;  Nielsen  and  Tahir,  2002). 
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Table  11.  Comparison  of  Average  Usability  Compliance  Score  by  Heuristic 


Heuristic 

Level* 

Mean 

13 

A 

0.82333333 

4 

A 

B 

0.75833333 

2 

A 

B 

0.73500000 

12 

A 

B 

0.71000000 

9 

A 

B 

0.70833333 

1 

A 

B 

0.68000000 

5 

A 

B 

0.67833333 

3 

A 

B 

0.63833333 

11 

A 

B 

C 

0.59833333 

7 

B 

C 

0.58333333 

6 

B 

C 

0.52333333 

10 

C 

0.39000000 

8 

C 

0.37166667 

*Levels  not  connected  by  the  same  letter  are  significantly 
different  (a  =  .05) 

Figure  5.  Average  Evaluator  Scores  for  Each  Case  by  Heuristic 


Mean  Usability  Compliance  Score  for  All  Cases. 

Taking  all  cases  into  consideration,  the  mean  usability  score  for  the  six  web  sites 
inspected  is  .63  with  all  six  web  sites  falling  into  the  scoring  range  of  .50  to  .80. 
According  to  the  established  criteria,  this  indicates  that  while  the  current  intranet  web 
sites  are  not  a  disaster,  they  are  bad  enough  that  isolated  modifications  to  individual  areas 
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will  not  suffice  (Felax,  2005;  Nielsen  and  Tahir,  2002).  Furthermore,  the  usability  score 
of  .63  is  very  close  to  the  findings  of  past  AFIT  research  where  Felax  calculated  a 
usability  score  of  .65  for  the  Air  Force  Knowledge  Now  web  site  (Felax,  2005)  and 
Kastenholz  calculated  a  usability  score  of  .6958  for  the  Automated  Civil  Engineer 
System  Personnel  Readiness  module  of  the  Automated  Civil  Engineer  System  Personnel 
Readiness  (Kastenholz,  2005). 

Investigative  Question  2:  Air  Force  Web  Site  Accessibility  Inspection  Results 

Accessibility  compliance  of  each  site  was  assessed  by  the  primary  researcher.  The 
raw  data  is  summarized  in  Table  12  and  the  following  sections  describe  how  well  the 
cases  comply  with  Section  508  mandatory  compliance  items. 

Analysis  of  Individual  Accessibility  Compliance  Scores  by  Guideline. 

As  shown  in  Table  12,  guidelines  1 194.22(j)  and  (m)  received  the  highest  scores, 
being  met  by  all  six  cases.  Additionally,  guidelines  1 194.22(d)  and  (g)  were  met  by  five 
of  the  cases.  This  indicates  that  the  cases  involved  in  this  study  adequately  addressed  the 
need  for  organizing  documents  so  they  are  readable  without  an  associated  style  sheet, 
properly  identified  row  and  column  headers,  avoided  screen  flicker  greater  than  2  Hz  and 
lower  than  55  Hz,  and  provided  associated  files  for  download  in  accessible  formats. 

In  contrast,  only  one  web  site  met  guideline  1 194.22(o)  by  providing  a  method  that 
pennits  users  to  skip  repetitive  navigation  links.  In  addition,  two  cases  failed  to 
adequately  address  guidelines  1 1 94.22(a),  provide  a  text  equivalent  for  all  non-text 
elements,  and  1 194.22(n),  use  forms  that  are  compatible  with  assistive  technologies. 
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Table  12.  Accessibility  Compliance  Scores  by  Case  and  Guideline 


Guideline 

A 

B 

C 

D 

E 

F 

Compliance 

Score 

1194.22  (a) 

1 

1 

1 

1 

0 

0 

0.67 

1194.22  (b) 

— 

0 

— 

0.00 

1194.22  (c) 

1 

1 

1 

1 

1 

0 

1194.22  (d) 

1 

1 

1 

0 

1 

1 

1194.22  (e) 

— 

— 

— 

— 

___ 

— 

— 

1194.22  (f) 

1 

1 

— 

— 

1 

— 

1.00 

1194.22  (g) 

1 

1 

1 

1 

1 

0 

0.83 

1194.22  (h) 

— 

— 

— 

— 

___ 

— 

1194.22  (i) 

— 

1 

— 

— 

___ 

— 

1.00 

1194.22  (j) 

1 

1 

1 

1 

1 

1 

1.00 

1194.22  (k) 

— 

— 

— 

— 

___ 

— 

— 

1194.22  (1) 

1 

1 

— 

1 

0 

1 

0.80 

1194.22  (m) 

1 

1 

1 

1 

1 

1 

1.00 

1194.22  (n) 

1 

0 

1 

1 

0 

1 

0.67 

1194.22  (o) 

0 

— 

0 

1 

0 

0 

0.20 

1194.22  (p) 

— 

— 

— 

— 

___ 

— 

— 

Compliance 

Score 

0.90 

0.82 

0.88 

0.89 

0.60 

0.56 

0.77 

(1)  Complies  (0)  Does  not  (--)  Not  applicable 

comply 

Guideline  1 194.22(b)  which  requires  synchronized  captions  for  multimedia  files  was  only 
applicable  to  one  case  and  that  case  did  not  comply  with  the  guideline. 

Analysis  Web  Site  Accessibility  Across  all  Cases. 

The  average  accessibility  compliance  score  across  all  web  sites  included  in  this 
study  is  .77.  Case  A  received  the  high  score  of  .90  and  Case  F  received  the  low  score  of 
.56.  In  all,  three  web  sites  scored  greater  than  .80  while  two  scored  lower.  Based  on  the 
established  grading  criteria,  the  three  sites  scoring  higher  than  .80  are  in  good  shape 
regarding  accessibility  and  should  consider  making  a  few  minor  fixes  to  areas  where  the 
site  violated  guidelines.  However,  the  two  cases  scoring  lower  than  .80  but  above  .50 
should  consider  starting  a  redesign  project  to  produce  a  new  web  site.  While  neither  of 
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these  sites  is  a  disaster,  they  are  bad  enough  that  isolated  modifications  to  individual 
areas  will  not  suffice  (Felax,  2005;  Nielsen  and  Tahir,  2002). 

Additional  inspection  of  the  data  reveals  that  the  accessibility  scores  calculated  in 
this  study  greatly  exceed  those  found  in  previous  studies  of  the  accessibility  of 
government  web  sites  (Ellison,  2004;  Felax,  2005;  Jackson-Sanborn  et  ah,  2002).  This 
may  be  due  to  better  compliance  by  the  sample  of  web  sites  used  in  this  study  or  to  a 
more  liberal  grading  standard  used  by  the  primary  researcher  in  this  study.  No 
assumption  is  made  about  which  explanation  is  more  accurate  and  it  is  noted  that  this 
issue  warrants  additional  research. 

Investigative  Question  3:  Analysis  of  the  Relationship  Between  Usability  and 
Accessibility  Scores 

Usability  and  accessibility  compliance  scores  obtained  from  IQ1  and  IQ2  were 
used  to  answer  this  investigative  question.  These  scores  were  charted  on  a  single  graph 
with  usability  compliance  scores  on  the  y-axis  and  accessibility  compliance  scores  on  the 
x-axis.  If  the  scores  are  related,  a  visible  pattern  should  appear  in  the  graph.  However  as 
shown  in  Figure  6,  there  appears  to  be  no  direct  linear  relationship  between  usability  and 
accessibility  compliance  scores.  Based  on  the  overall  compliance  scores  for  each  web 
site,  heuristic  compliance  scores  are  not  correlated  with  accessibility  compliance  scores. 

Investigative  Question  4:  Analysis  of  Military  Policies 

A  list  of  military  guidance  documents  relevant  to  web  site  development, 
maintenance,  and  use  was  compiled  and  reviewed  for  overlap  with  common  practice 
usability  heuristics  and  mandatory  Section  508  compliance  guidelines.  This  review 
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Figure  6.  Graph  of  Usability  Scores  as  a  Function  of  Accessibility  Scores 


identified  eleven  military  guidance  documents  with  which  compliance  is  mandatory  for 
Air  Force  organizations.  Of  these  eleven  documents,  three  are  Air  Force  publications 
focused  on  Internet  use  and  web  site  development,  maintenance,  and  use.  Six  include 
guidance  that  is  mandatory  for  all  Air  Force  web  sites  but  does  not  elaborate  on  site 
development,  maintenance,  or  use.  The  remaining  two  documents  are  DoD  publications 
that  provide  mandatory  guidance  that  is  relevant  to  web  sites.  The  results  of  this  review 
are  show  in  Tables  13  and  14. 

Of  the  thirteen  common  practice  heuristics  used  in  this  study,  five  were  found  to 
be  represented  in  mandatory  military  guidance,  a  total  of  38  percent.  In  total,  these  five 
heuristics  were  mentioned  12  times  with  privacy  receiving  eight  of  those  mentions.  One 
document,  Air  Force  Instruction  33-129  Web  Management  and  Internet  Use,  contained 
references  to  four  of  the  five  heuristics  mentioned. 

Meanwhile,  statements  mandating  Section  508  compliance  were  found  four  times 
in  these  same  military  guidelines.  However  only  six  of  the  sixteen  individual 
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Table  13.  Number  of  Occurrences  of  Each  Heuristic  in  Military  Guidance 


Publication 

Heuristic 

1  2  3  4  5  6  7  8  9  10  11  12  13 

Total 

Air  Force  Mandatory  Internet  Guidance 

LJ 

JL 

LJ 

LJ 

El 

m 

B 

B 

4 

JJ 

JJ 

JJ 

B 

1 

Air  Force  Mandatory  non-Web  Specific  Guidance 

□ 

B 

1 

1  AFMAN  33-326  ( 

KIM 

LJ 

1 

B 

1 

AFMAN  37-104 

B 

1 

AFPD  37-1 

B 

1 

DoD  Reg  5400.7/AF  Sup 

B 

1 

DoD  Mandatory  Guidance 

DODD  8000.1 

rj 

LJ 

□ 

JJ 

jj 

jj 

jj 

jj 

JJ 

JL 

Web  Site  Admin  Policies  and  Procedures 

LJ 

LJ 

LJ 

JL 

tj 

LJ 

LJ 

JL 

JL 

B 

1 

Available  Reference  but  not  Mandatory  Guidance 

AF  Content  Publishing  Guide  for  Publishing  Center  vl.l 

m 

B 

B 

B 

JJ 

B 

B 

B 

JJ 

LJ 

JJ 

7 

AF  Portal  Style  Guide 

B 

jj 

JJ 

B 

tJ 

B 

JJ 

JJ 

tJ 

B 

M 

4 

#  Occurrences  in  Mandatory  Guidance 

B 

□ 

B 

□ 

B 

B 

B 

B 

□ 

B 

B 

□ 

□ 

Table  14.  Number  of  Mentions  of  Mandatory  Section  508  Compliance  and  Occurrences  of 
_ Individual  Section  508  Guidelines  in  Military  Guidance _ 


Publication 

■n 

Specific  1194.22  Paragraph 

Total 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

l 

m 

n 

0 

p 

Individual 

Air  Force  Mandatory  Internet  Guidance 

AFI  33-117 

X 

X 

0 

X 

□ 

m 

B 

B 

B 

B 

6 

Air  Force  Mandatory  non-Web  Specific  Guidance 

0 

AFMAN  33-326 

0 

0 

AFMAN  37-104 

0 

AFPD  37-1 

DoD  Reg  5400.7/AF  Sup 

0 

DoD  Mandatory  Guidance 

DODD  8000.1 

X 

Web  Site  Admin  Policies  and  Procedures 

0 

Available  Reference  but  not  Mandatory  Guidance 

AF  Content  Publishing  Guide  for  Publishing  Center  vl.l 

X 

b 

B 

B 

B 

m 

B 

6 

AF  Portal  Style  Guide 

0 

#  Occurrences  in  Mandatory  Guidance 

4 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

fl 

B 

accessibility  compliance  items  mandated  by  the  Access  Board  were  expressly  mentioned, 
a  total  of  38  percent.  All  six  of  these  mentions  occurred  in  the  same  document,  Air  Force 
Instruction  35-101  Public  Affairs  Policies  and  Procedures. 

Based  on  these  findings,  the  majority  of  common  practice  usability  guidelines  are 
not  found  in  mandatory  military  guidance.  Likewise,  the  majority  of  specific 
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accessibility  compliance  guidelines  outlined  by  the  Access  Board  are  absent  from 
military  guidance  documents.  However  the  overarching  principle  that  web  sites  must  be 
Section  508  compliant  appears  to  be  adequately  addressed,  being  clearly  stated  in  four 
military  policy  documents,  three  of  which  are  specific  to  web  site  development, 
maintenance,  and  use. 

Investigative  Question  5:  Analysis  of  the  Relationship  Between  Military  Guidance 
and  Compliance  Scores 

The  number  of  times  that  common  practice  usability  heuristics  and  Section  508 
compliance  is  discussed  in  military  guidance  is  compared  with  the  compliance  scores  to 
answer  this  investigative  question.  In  the  first  step,  data  was  charted  on  a  single  graph 
with  mean  compliance  scores  for  each  heuristic  (including  one  data  point  representing 
Section  508  compliance)  on  the  y-axis  and  the  number  of  occurrences  in  the  military 
guidelines  on  the  x-axis.  If  the  scores  are  related  to  the  number  of  occurrences  in  the 
military  guidelines,  a  pattern  should  appear  in  the  graph.  As  shown  in  Figure  7,  it 
appears  that  higher  occurrences  of  a  given  heuristic  are  correlated  with  higher  compliance 
scores  for  that  same  heuristic. 

To  provide  another  perspective,  heuristics  were  then  divided  into  two  groups. 

One  group  consisted  of  all  heuristics  not  represented  in  military  guidance  and  the  other 
group  consisted  of  all  heuristics  that  are  represented  in  military  guidance.  Again, 
accessibility  compliance  is  included  as  a  single  data  point.  Figure  8  reveals  the 
dichotomy  in  compliance  scores  between  these  groups,  showing  that  mean  scores  for 
heuristics  represented  in  military  guidance  are  higher  than  for  non-represented  heuristics. 
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Figure  7.  Graph  of  Compliance  Scores  as  a  Function  of  the  Number  of  Occurrences  of 
Each  Fieuristic  in  Military  Guidance 


Figure  8.  Graph  of  Compliance  Scores  Grouped  by  Occurrences  in 
Mandatory  Military  Guidance 


Based  on  these  comparisons,  it  appears  that  a  correlation  exists  between  the 
existence  of  heuristics  in  military  guidance  and  the  compliance  scores  associated  with 
those  heuristics. 


Ad  Hoc  Analysis 

Although  additional  analysis  is  not  required  to  answer  the  investigative  questions, 
several  findings  warrant  additional  analysis.  This  ad  hoc  analysis  looks  at  what  might 
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happen  to  the  compliance  scores  if  the  best  or  worst  heuristics  were  removed  from  the 
analysis  and  what  effect  additional  mandatory  guidance  could  have  on  the  overall  quality 
of  military  guidance. 

IQ  1  revealed  that  privacy  was  the  heuristic  best  addressed  by  the  sites  studied  and 
that  flexibility  and  efficiency  of  use  and  help  and  documentation  were  the  worst  areas.  If 
privacy  is  removed  from  the  analysis,  the  overall  compliance  score  drops  from  .63  to  .61. 
On  the  other  hand  removing  either  of  the  two  lowest  scoring  heuristics  raises  the  overall 
compliance  score  from  .63  to  .65  while  removing  both  raises  the  overall  score  to  .68. 

This  analysis  shows  that  neither  the  highest  or  lowest  scoring  heuristics  are  strongly 
influencing  the  overall  mean  usability  score.  Without  the  highest  score  the  mean  does 
not  fall  below  the  next  cutoff  of  .50  nor  does  it  rise  above  .80  if  the  lowest  scores  are 
removed.  This  reinforces  the  finding  that  isolated  modifications  to  individual  areas  will 
not  suffice  to  raise  the  average  usability  of  these  sites  to  an  acceptable  level. 

IQ4  found  that  only  five  of  Nielsen’s  thirteen  usability  heuristics  were  found  in 
military  guidance  documents.  Aside  from  the  mandatory  guidance  already  discussed,  the 
review  of  military  guidance  also  identified  two  additional  documents  that  are  relevant  to 
some  Air  Force  intranet  web  sites  (also  detailed  in  Appendix  B).  These  documents  were 
specifically  targeted  at  sites  appearing  on  the  Air  Force  Portal  and,  although  compliance 
with  these  guidelines  is  not  mandatory  for  non-Portal  web  sites,  these  documents  could 
serve  as  an  additional  reference  for  non-Portal  sites.  These  two  guidelines  capture  eight 
of  Nielsen’s  heuristics,  four  of  which  are  not  otherwise  found  in  the  military  guidance. 
Simply  adding  these  documents  to  the  list  of  mandatory  guidance  would  raise  the  number 
of  usability  heuristics  contained  in  military  policy  from  38  percent  to  69  percent. 
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In  addition,  these  two  documents  also  provide  specific  guidance  for  six  Section 
508  compliance  items,  four  of  which  are  unique  from  those  found  already  present  in 
mandatory  guidance.  Adding  these  documents  to  the  list  of  mandatory  guidance  would 
raise  the  average  number  of  Section  508  compliance  items  contained  in  military  guidance 
from  38  percent  to  63  percent.  These  findings  highlight  the  fact  that  more  robust  military 
guidance  could  be  easily  developed  by  simply  integrating  the  principles  found  in  those 
two  documents  into  mandatory  compliance  documents. 

Chapter  Summary 

This  chapter  summarized  the  findings  of  this  study’s  five  investigative  questions. 
The  average  compliance  rate  for  the  web  sites  evaluated  was  found  to  be  63  percent.  The 
average  accessibility  score  was  77  percent.  Furthermore,  accessibility  compliance  was 
found  to  have  no  correlation  with  usability  compliance.  A  review  of  mandatory  military 
guidance  found  that  only  38  percent  of  usability  heuristics  and  38  percent  of  mandatory 
accessibility  requirements  are  captured  in  military  guidance.  In  addition,  a  correlation 
was  found  between  the  existence  of  usability  and  accessibility  principles  in  military 
guidance  and  compliance  scores  for  those  principles. 

In  answer  to  the  primary  research  question,  Air  Force  intranet  web  sites  in  this 
study  did  not  adequately  comply  with  basic  common  practice  usability  guidelines  and 
compliance  with  mandatory  Section  508  compliance  items  varied  among  sites  with  four 
of  six  sites  receiving  acceptable  accessibility  compliance  scores. 
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V.  Discussion 


Introduction 

This  chapter  provides  discussion  on  the  findings  presented  in  the  Chapter  IV  and 
makes  recommendations  based  on  those  findings.  Limitations  are  summarized  and 
suggests  for  future  research  are  offered. 

Usability  Recommendations 

This  research  found  substantial  usability  problems  with  each  web  site  evaluated. 
The  cases  selected  for  this  research  are  representative  of  the  types  of  applications  being 
migrated  to  web  interfaces.  Therefore  it  is  important  to  understand  what  lessons  can  be 
learned  from  this  study  to  improve  these  sites  and  prevent  other  web  applications  from 
suffering  similar  usability  problems. 

Heuristic  evaluations  are  traditionally  recognized  as  a  simple  approach  to  identify 
the  highest  priority  areas  on  which  to  commit  limited  organizational  resources.  However 
it  is  critical  to  realize  that  isolated  changes  and  modifications  will  not  suffice  to  correct 
the  usability  problems  identified  by  this  research.  Consequently  individual 
recommendations  for  quick  fixes  to  specific  problems  are  not  appropriate  and  are  not 
provided.  Although  the  lowest  compliance  item(s)  for  each  usability  heuristic  is 
identified  in  Chapter  IV,  changes  must  be  implemented  as  part  of  a  wider  redesign  effort 
focused  on  improving  all  aspects  of  usability.  However,  it  is  noted  that  extra  attention 
should  be  focused  on  flexibility  and  efficiency  of  use  and  help  and  documentation,  the 
two  usability  heuristics  which  received  the  worst  compliance  scores.  Furthermore  the 
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comparatively  high  compliance  rates  found  for  privacy  raise  concerns  that  the  resources 
and  attention  dedicated  to  privacy  may  have  come  at  the  expense  of  other  aspects  of 
usability.  Organizations  are  therefore  cautioned  to  carefully  evaluate  the  optimal  balance 
for  resource  allotment  to  each  usability  area. 

Accessibility  Recommendations 

This  study  identified  only  one  Section  508  compliance  item  that  was  consistently 
overlooked  by  nearly  all  web  sites.  This  item,  provide  a  method  that  permits  users  to 
skip  repetitive  navigational  links,  is  a  simple  update  that  must  be  made  to  all 
noncompliant  web  sites.  Based  on  the  other  accessibility  findings,  organizations  are 
encouraged  to  ensure  that  a  text  equivalent  exists  for  all  non-text  elements  and  redesign 
all  fonns  to  be  fully  compatible  with  assistive  technologies.  The  four  cases  with  sites 
scoring  above  .80  should  review  all  compliance  items  and  take  necessary  actions  to 
correct  individual  instances  of  non-conformance.  The  two  cases  with  sites  that  received 
overall  accessibility  scores  below  .80  should  initiate  a  site  redesign  focused  on  meeting 
all  mandatory  accessibility  requirements. 

Discussion  About  the  Relationship  Between  Usability  and  Accessibility 

This  study  found  no  evidence  of  a  significant  correlation  between  Nielsen’s 
usability  heuristics  and  government  mandated  accessibility  compliance  items.  This 
research  adds  to  the  existing  body  of  knowledge  but  should  not  be  taken  out  of  context. 
Section  508  compliance  items  largely  address  technical  accessibility  issues.  Therefore,  it 
is  most  appropriate  to  state  that  this  study  found  no  evidence  of  a  correlation,  either 
positive  or  negative,  between  usability  and  technical  accessibility. 
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Recommendations  for  Military  Web  Site  Guidance 

This  research  supports  the  findings  by  Kastenholz  that  compliance  with  military 
guidelines  is  not  sufficient  to  ensure  creation  of  a  usable  web  site  (Kastenholz,  2005). 
Although  basic  usability  and  accessibility  principles  are  not  widely  captured  in  military 
guidance,  those  that  do  appear  in  the  guidance  are  correlated  with  higher  compliance 
scores.  This  appears  to  be  especially  true  for  privacy  which  is  a  widely  documented 
military  concern  and  is  the  usability  heuristic  that  received  the  highest  compliance  scores 
in  this  study.  These  findings  support  the  need  for  the  Air  Force  to  develop  a  single, 
overarching  web  site  style  guide  that  adequately  captures  basic  HCI  design  principles  and 
fully  documents  the  requirements  for  Section  508  compliance. 

Research  limitations 

This  study  was  designed  to  address  the  primary  limitation  of  past  research  by 
using  a  minimum  of  three  evaluators  per  site  and  providing  each  evaluator  with  usability 
training  (Felax,  2005;  Kastenholz,  2005).  However,  some  limitations  exist. 

One  limitation  is  the  use  of  students  as  usability  experts.  Although  training  was 
provided,  it  is  unrealistic  to  assert  that  their  skills  are  at  the  same  level  as  professional 
usability  experts  who  have  years  of  HCI  experience.  The  most  likely  outcome  of 
substituting  students  for  experts  is  that  students  will  not  identify  as  many  usability 
problems  as  experts  would.  This  implies  that  compliance  scores  would  be  lower  if 
experts  were  used. 

Another  limitation  is  that  the  use  of  a  single  accessibility  evaluator  provides 
weaker  support  for  accessibility  findings  than  would  the  use  of  multiple  evaluators. 
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However,  accessibility  evaluations  are  more  technical  in  nature  and  the  additional 
evaluators  available  for  this  effort  did  not  possess  the  technical  skills  required  to  conduct 
accessibility  inspections. 

Two  other  limitations  are  the  limited  availability  of  relevant,  web  specific  military 
guidance  and  the  unknown  degree  to  which  individual  military  units  comply  with  that 
guidance.  This  study  focused  solely  on  general  guidance  such  as  DoD  Regulations  and 
Air  Force  Instructions  that  were  readily  available  for  evaluation.  This  approach  excluded 
the  large  body  of  program  specific  design  documents  used  for  development  and 
acquisition  of  web-based  applications.  Likewise,  organizational  guidance  produced  for 
use  on  at  the  local  unit  level  was  not  included.  Furthermore,  although  military  guidance 
can  easily  be  qualified  as  mandatory  or  optional,  cases  were  not  questioned  to  determine 
their  knowledge  or  use  of  mandatory  guidance. 

Recommendations  for  Future  Research 

The  relatively  high  compliance  rate  for  privacy  may  indicate  that  an  inverse 
relationship  exits  between  privacy  as  implemented  by  Air  Force  web  sites  and  the  other 
heuristics.  One  possibility  is  that  privacy  is  given  preferential  treatment  and  allowed  to 
take  precedence  over  other  aspects  of  usability  when  conflicts  exist.  Another  possibility 
is  that  more  resources  are  spent  on  privacy  leaving  fewer  resources  for  other  aspects  of 
usability.  Future  research  could  investigate  possible  covariance  between  privacy  and  the 
other  usability  heuristics. 

Regarding  accessibility,  future  research  may  benefit  from  the  use  of  an  automated 
accessibility  inspection  tool.  The  findings  of  an  automated  tool  would  be  more  objective 
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and  arguably  more  rigorous  by  ensuring  a  more  consistent  evaluation  across  web  sites 
than  is  possible  with  a  human  evaluator.  Additionally,  an  automated  tool  could  easily 
evaluate  accessibility  against  Section  508  compliance  items  and  against  WAI  Web 
Content  Accessibility  Guidelines  to  provide  a  deeper  understanding  of  overall 
accessibility. 

A  study  should  be  conducted  to  perfonn  a  comprehensive  review  of  all  military 
web  site  development  guidance  and  handbooks  across  programs  and  functional  areas. 
Such  a  review  should  include  the  design  documents  used  for  system  development  and 
acquisition  as  well  as  guidelines  produced  below  the  Air  Force  level  such  as  Major 
Command  and  local  unit  guidance.  This  review  could  also  follow  Ratners’s  work  by 
evaluating  the  military  guidance  against  the  270  HTML-relevant  recommendations  she 
identified  in  HCI  guidelines  (Ratner  et  ah,  1996). 

Although  the  applicability  of  TAM  is  strongly  supported  in  web  based  and 
mandatory  adoption  environments,  the  question  of  exactly  how  much  usability  is 
sufficient  has  not  been  definitively  answered.  Indeed,  there  is  ongoing  discussion  in  both 
academic  and  practitioner  circles  as  to  how  much  should  be  invested  in  usability. 
Therefore,  future  research  could  compare  usability  compliance  to  a  web  site  acceptance 
construct  to  identify  trends  and  attempt  to  quantify  the  level  of  usability  required  for 
specific  web  site  functions. 

Chapter  Summary 

This  chapter  provided  a  discussion  of  the  results  documented  in  Chapter  IV.  The 
web  sites  evaluated  by  this  study  faired  poorly  against  Nielsen’s  usability  heuristics  but 
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most  met  mandatory  accessibility  standards.  These  sites  should  consider  redesign 
projects  to  develop  more  usable  sites  and  address  any  outstanding  accessibility  problems. 
The  results  of  this  study  present  a  learning  opportunity  for  Air  Force  organizations 
responsible  for  web  site  design  and  maintenance  and  provide  evidence  of  the  value  of 
creating  a  single,  robust,  program-independent  Air  Force  web  style  guide  that  adequately 
captures  basic  HCI  design  principles  and  fully  documents  the  requirements  for  Section 
508  compliance. 
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Appendix  A.  Usability  Heuristic  Checklist 


The  following  checklist  comes  from  Pierotti  and  is  based  on  Nielsen’s  usability 
heuristics  (Nielsen  and  Mack,  1994;  Pierotti,  1995). 


1.  Visibility  of  System  Status 

The  system  should  always  keep  users  informed  about  what  is  going  on,  through 
appropriate  feedback  within  reasonable  time. 


# 

Review  Checklist 

1.1 

Does  every  display  begin  with  a  title  or  header  that  describes  screen  contents? 

1.2 

Is  there  a  consistent  icon  design  scheme  and  stylistic  treatment  across  the  system? 

1.3 

Is  a  single,  selected  icon  clearly  visible  when  surrounded  by  unselected  icons? 

1.4 

Do  menu  instructions,  prompts,  and  error  messages  appear  in  the  same  place(s)  on  each  menu? 

1.5 

In  multi-page  data  entry  screens,  is  each  page  labeled  to  show  its  relation  to  others? 

1.6 

If  overtype  and  insert  mode  are  both  available,  is  there  a  visible  indication  of  which  one  the 
user  is  in? 

1.7 

If  pop-up  windows  are  used  to  display  error  messages,  do  they  allow  the  user  to  see  the  field  in 
error? 

1.8 

Is  there  some  form  of  system  feedback  for  every  operator  action? 

1.9 

After  the  user  completes  an  action  (or  group  of  actions),  does  the  feedback  indicate  that  the 
next  group  of  actions  can  be  started? 

1.10 

Is  there  visual  feedback  in  menus  or  dialog  boxes  about  which  choices  are  selectable? 

1.11 

Is  there  visual  feedback  in  menus  or  dialog  boxes  about  which  choice  the  cursor  is  on  now? 

1.12 

If  multiple  options  can  be  selected  in  a  menu  or  dialog  box,  is  there  visual  feedback  about 
which  options  are  already  selected? 

1.13 

Is  there  visual  feedback  when  objects  are  selected  or  moved? 

1.14 

Is  the  current  status  of  an  icon  clearly  indicated? 

1.15 

Is  there  feedback  when  function  keys  are  pressed? 

1.16 

If  there  are  observable  delays  (greater  than  fifteen  seconds)  in  the  system’s  response  time,  is 
the  user  kept  informed  of  the  system's  progress? 

1.17 

Are  response  times  appropriate  to  the  task? 

1.18 

Typing,  cursor  motion,  mouse  selection:  50-1  50  milliseconds 

1.19 

Simple,  frequent  tasks:  less  than  1  second 

1.20 

Common  tasks:  2-4  seconds 

1.21 

Complex  tasks:  8-12  seconds 
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1.22 

Are  response  times  appropriate  to  the  user's  cognitive  processing? 

1.23 

Continuity  of  thinking  is  required  and  information  must  be  remembered  throughout 
several  responses:  less  than  two  seconds. 

1.24 

High  levels  of  concentration  aren't  necessary  and  remembering  information  is  not 
required:  two  to  fifteen  seconds. 

1.25 

Is  the  menu-naming  terminology  consistent  with  the  user's  task  domain? 

1.26 

Does  the  system  provide  visibility:  that  is,  by  looking,  can  the  user  tell  the  state  of  the  system 
and  the  alternatives  for  action? 

1.27 

Do  GUI  menus  make  obvious  which  item  has  been  selected? 

1.28 

Do  GUI  menus  make  obvious  whether  de-selection  is  possible? 

1.29 

If  users  must  navigate  between  multiple  screens,  does  the  system  use  context  labels,  menu 
maps,  and  place  markers  as  navigational  aids? 

2.  Match  Between  System  and  the  Real  World 

The  system  should  speak  the  users'  language,  with  words,  phrases  and  concepts 
familiar  to  the  user,  rather  than  system-oriented  terms.  Follow  real-world  conventions, 
making  infonnation  appear  in  a  natural  and  logical  order. 


# 

Review  Checklist 

2.1 

Are  icons  concrete  and  familiar? 

2.2 

Are  menu  choices  ordered  in  the  most  logical  way,  given  the  user,  the  item  names,  and  the  task 
variables? 

2.3 

If  there  is  a  natural  sequence  to  menu  choices,  has  it  been  used? 

2.4 

Do  related  and  interdependent  fields  appear  on  the  same  screen? 

2.5 

If  shape  is  used  as  a  visual  cue,  does  it  match  cultural  conventions? 

2.6 

Do  the  selected  colors  correspond  to  common  expectations  about  color  codes? 

2.7 

When  prompts  imply  a  necessary  action,  are  the  words  in  the  message  consistent  with  that 
action? 

2.8 

Do  keystroke  references  in  prompts  match  actual  key  names? 

2.9 

On  data  entry  screens,  are  tasks  described  in  terminology  familiar  to  users? 

2.10 

Are  field-level  prompts  provided  for  data  entry  screens? 

2.11 

For  question  and  answer  interfaces,  are  questions  stated  in  clear,  simple  language? 

2.12 

Do  menu  choices  fit  logically  into  categories  that  have  readily  understood  meanings? 

2.13 

Are  menu  titles  parallel  grammatically? 

2.14 

Does  the  command  language  employ  user  jargon  and  avoid  computer  jargon? 

2.15 

Are  command  names  specific  rather  than  general? 

2.16 

Does  the  command  language  allow  both  full  names  and  abbreviations? 

2.17 

Are  input  data  codes  meaningful? 

2.18 

Have  uncommon  letter  sequences  been  avoided  whenever  possible? 

2.19 

Does  the  system  automatically  enter  leading  or  trailing  spaces  to  align  decimal  points? 
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2.20 

Does  the  system  automatically  enter  a  dollar  sign  and  decimal  for  monetary  entries? 

2.21 

Does  the  system  automatically  enter  commas  in  numeric  values  greater  than  9999? 

2.22 

Do  GUI  menus  offer  activation:  that  is,  make  obvious  how  to  say  “now  do  it"? 

2.23 

Has  the  system  been  designed  so  that  keys  with  similar  names  do  not  perform  opposite  (and 
potentially  dangerous)  actions? 

2.24 

Are  function  keys  labeled  clearly  and  distinctively,  even  if  this  means  breaking  consistency 
rules? 

3.  User  Control  and  Freedom 

Users  should  be  free  to  select  and  sequence  tasks  (when  appropriate),  rather  than 
having  the  system  do  this  for  them.  Users  often  choose  system  functions  by  mistake  and 
will  need  a  clearly  marked  “emergency  exit”  to  leave  the  unwanted  state  without  having 
to  go  through  an  extended  dialogue.  Users  should  make  their  own  decisions  (with  clear 
information)  regarding  the  costs  of  exiting  current  work.  The  system  should  support 
undo  and  redo. 


# 

Review  Checklist 

3.1 

If  setting  up  windows  is  a  low-frequency  task,  is  it  particularly  easy  to  remember? 

3.2 

In  systems  that  use  overlapping  windows,  is  it  easy  for  users  to  rearrange  windows  on  the 
screen? 

3.3 

In  systems  that  use  overlapping  windows,  is  it  easy  for  users  to  switch  between  windows? 

3.4 

When  a  user's  task  is  complete,  does  the  system  wait  for  a  signal  from  the  user  before 
processing? 

3.5 

Can  users  type-ahead  in  a  system  with  many  nested  menus? 

3.6 

Are  users  prompted  to  confirm  commands  that  have  drastic,  destructive  consequences? 

3.7 

Is  there  an  "undo"  function  at  the  level  of  a  single  action,  a  data  entry,  and  a  complete  group  of 
actions? 

3.8 

Can  users  cancel  out  of  operations  in  progress? 

3.9 

Are  character  edits  allowed  in  commands? 

3.10 

Can  users  reduce  data  entry  time  by  copying  and  modifying  existing  data? 

3.11 

Are  character  edits  allowed  in  data  entry  fields? 

3.12 

If  menu  lists  are  long  (more  than  seven  items),  can  users  select  an  item  either  by  moving  the 
cursor  or  by  typing  a  mnemonic  code? 

3.13 

If  the  system  uses  a  pointing  device,  do  users  have  the  option  of  either  clicking  on  menu  items 
or  using  a  keyboard  shortcut? 

3.14 

Are  menus  broad  (many  items  on  a  menu)  rather  than  deep  (many  menu  levels)? 
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3.15 

If  the  system  has  multiple  menu  levels,  is  there  a  mechanism  that  allows  users  to  go  back  to 
previous  menus? 

3.16 

If  users  can  go  back  to  a  previous  menu,  can  they  change  their  earlier  menu  choice? 

3.17 

Can  users  move  forward  and  backward  between  fields  or  dialog  box  options? 

3.18 

If  the  system  has  multi-page  data  entry  screens,  can  users  move  backward  and  forward  among 
all  the  pages  in  the  set? 

3.19 

If  the  system  uses  a  question  and  answer  interface,  can  users  go  back  to  previous  questions  or 
skip  forward  to  later  questions? 

3.20 

Do  function  keys  that  can  cause  serious  consequences  have  an  undo  feature? 

3.21 

Can  users  easily  reverse  their  actions? 

3.22 

If  the  system  allows  users  to  reverse  their  actions,  is  there  a  retracing  mechanism  to  allow  for 
multiple  undos? 

3.23 

Can  users  set  their  own  system,  session,  file,  and  screen  defaults? 

4.  Consistency  and  Standards 

Users  should  not  have  to  wonder  whether  different  words,  situations,  or  actions 
mean  the  same  thing.  Follow  platform  conventions. 


# 

Review  Checklist 

4.1 

Have  industry  or  company  formatting  standards  been  followed  consistently  in  all  screens 
within  a  system? 

4.2 

Has  a  heavy  use  of  all  uppercase  letters  on  a  screen  been  avoided? 

4.3 

Do  abbreviations  not  include  punctuation? 

4.4 

Are  integers  right-justified  and  real  numbers  decimal-aligned? 

4.5 

Are  icons  labeled? 

4.6 

Are  there  no  more  than  twelve  to  twenty  icon  types? 

4.7 

Are  there  salient  visual  cues  to  identify  the  active  window? 

4.8 

Does  each  window  have  a  title? 

4.9 

Are  vertical  and  horizontal  scrolling  possible  in  each  window? 

4.10 

Does  the  menu  structure  match  the  task  structure? 

4.11 

Have  industry  or  company  standards  been  established  for  menu  design,  and  are  they  applied 
consistently  on  all  menu  screens  in  the  system? 

4.12 

Are  menu  choice  lists  presented  vertically? 

4.13 

If  "exit"  is  a  menu  choice,  does  it  always  appear  at  the  bottom  of  the  list? 

4.14 

Are  menu  titles  either  centered  or  left-justified? 

4.15 

Are  menu  items  left-justified,  with  the  item  number  or  mnemonic  preceding  the  name? 

4.16 

Do  embedded  field-level  prompts  appear  to  the  right  of  the  field  label? 

4.17 

Do  on-line  instructions  appear  in  a  consistent  location  across  screens? 

4.18 

Are  field  labels  and  fields  distinguished  typographically? 

4.19 

Are  field  labels  consistent  from  one  data  entry  screen  to  another? 

4.20 

Are  fields  and  labels  left -justified  for  alpha  lists  and  right-justified  for  numeric  lists? 
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4.21 

Do  field  labels  appear  to  the  left  of  single  fields  and  above  list  fields? 

4.22 

Are  attention-getting  techniques  used  with  care? 

4.23 

Intensity:  two  levels  only 

4.24 

Size:  up  to  four  sizes 

4.25 

Font:  up  to  three 

4.26 

Blink:  two  to  four  hertz 

4.27 

Color:  up  to  four  (additional  colors  for  occasional  use  only) 

4.28 

Sound:  soft  tones  for  regular  positive  feedback,  harsh  for  rare  critical  conditions 

4.29 

Are  attention-getting  techniques  used  only  for  exceptional  conditions  or  for  time-dependent 
information? 

4.30 

Are  there  no  more  than  four  to  seven  colors,  and  are  they  far  apart  along  the  visible  spectrum? 

4.31 

Is  a  legend  provided  if  color  codes  are  numerous  or  not  obvious  in  meaning? 

4.32 

Have  pairings  of  high-chroma,  spectrally  extreme  colors  been  avoided? 

4.33 

Are  saturated  blues  avoided  for  text  or  other  small,  thin  line  symbols? 

4.34 

Is  the  most  important  information  placed  at  the  beginning  of  the  prompt? 

4.35 

Are  user  actions  named  consistently  across  all  prompts  in  the  system? 

4.36 

Are  system  objects  named  consistently  across  all  prompts  in  the  system? 

4.37 

Do  field-level  prompts  provide  more  information  than  a  restatement  of  the  field  name? 

4.38 

For  question  and  answer  interfaces,  are  the  valid  inputs  for  a  question  listed? 

4.39 

Are  menu  choice  names  consistent,  both  within  each  menu  and  across  the  system,  in 
grammatical  style  and  terminology? 

4.40 

Does  the  structure  of  menu  choice  names  match  their  corresponding  menu  titles? 

4.41 

Are  commands  used  the  same  way,  and  do  they  mean  the  same  thing,  in  all  parts  of  the 
system? 

4.42 

Does  the  command  language  have  a  consistent,  natural,  and  mnemonic  syntax? 

4.43 

Do  abbreviations  follow  a  simple  primary  rule  and,  if  necessary,  a  simple  secondary  rule  for 
abbreviations  that  otherwise  would  be  duplicates? 

4.44 

Is  the  secondary  rule  used  only  when  necessary? 

4.45 

Are  abbreviated  words  all  the  same  length? 

4.46 

Is  the  structure  of  a  data  entry  value  consistent  from  screen  to  screen? 

4.47 

Is  the  method  for  moving  the  cursor  to  the  next  or  previous  field  consistent  throughout  the 
system? 

4.48 

If  the  system  has  multi-page  data  entry  screens,  do  all  pages  have  the  same  title? 

4.49 

If  the  system  has  multi-page  data  entry  screens,  does  each  page  have  a  sequential  page 
number? 

4.50 

Does  the  system  follow  industry  or  company  standards  for  function  key  assignments? 

4.51 

Are  high-value,  high-chroma  colors  used  to  attract  attention? 

5.  Help  Users  Recognize,  Diagnose,  and  Recover  From  Errors 

Error  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution. 
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# 

Review  Checklist 

5.1 

Is  sound  used  to  signal  an  error? 

5.2 

Are  prompts  stated  constructively,  without  overt  or  implied  criticism  of  the  user? 

5.3 

Do  prompts  imply  that  the  user  is  in  control? 

5.4 

Are  prompts  brief  and  unambiguous. 

5.5 

Are  error  messages  worded  so  that  the  system,  not  the  user,  takes  the  blame? 

5.6 

If  humorous  error  messages  are  used,  are  they  appropriate  and  inoffensive  to  the  user 
population? 

5.7 

Are  error  messages  grammatically  correct? 

5.8 

Do  error  messages  avoid  the  use  of  exclamation  points? 

5.9 

Do  error  messages  avoid  the  use  of  violent  or  hostile  words? 

5.10 

Do  error  messages  avoid  an  anthropomorphic  tone? 

5.11 

Do  all  error  messages  in  the  system  use  consistent  grammatical  style,  form,  terminology,  and 
abbreviations? 

5.12 

Do  messages  place  users  in  control  of  the  system? 

5.13 

Does  the  command  language  use  normal  action-object  syntax? 

5.14 

Does  the  command  language  avoid  arbitrary,  non-English  use  of  punctuation,  except  for 
symbols  that  users  already  know? 

5.15 

If  an  error  is  detected  in  a  data  entry  field,  does  the  system  place  the  cursor  in  that  field  or 
highlight  the  error? 

5.16 

Do  error  messages  inform  the  user  of  the  error's  severity? 

5.17 

Do  error  messages  suggest  the  cause  of  the  problem? 

5.18 

Do  error  messages  provide  appropriate  semantic  information? 

5.19 

Do  error  messages  provide  appropriate  syntactic  information? 

5.20 

Do  error  messages  indicate  what  action  the  user  needs  to  take  to  correct  the  error? 

5.21 

If  the  system  supports  both  novice  and  expert  users,  are  multiple  levels  of  error-message  detail 
available? 

6.  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a 
problem  from  occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or 
check  for  them  and  present  users  with  a  confirmation  option  before  they  commit  to  the 
action. 


# 

Review  Checklist 

6.1 

If  the  database  includes  groups  of  data,  can  users  enter  more  than  one  group  on  a  single 
screen? 

6.2 

Have  dots  or  underscores  been  used  to  indicate  field  length? 
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6.3 

Is  the  menu  choice  name  on  a  higher-level  menu  used  as  the  menu  title  of  the  lower-level 
menu? 

6.4 

Are  menu  choices  logical,  distinctive,  and  mutually  exclusive? 

6.5 

Are  data  inputs  case-blind  whenever  possible? 

6.6 

If  the  system  displays  multiple  windows,  is  navigation  between  windows  simple  and  visible? 

6.7 

Are  the  function  keys  that  can  cause  the  most  serious  consequences  in  hard-to-reach  positions? 

6.8 

Are  the  function  keys  that  can  cause  the  most  serious  consequences  located  far  away  from 
low-consequence  and  high-use  keys? 

6.9 

Has  the  use  of  qualifier  keys  been  minimized? 

6.10 

If  the  system  uses  qualifier  keys,  are  they  used  consistently  throughout  the  system? 

6.11 

Does  the  system  prevent  users  from  making  errors  whenever  possible? 

6.12 

Does  the  system  warn  users  if  they  are  about  to  make  a  potentially  serious  error? 

6.13 

Does  the  system  intelligently  interpret  variations  in  user  commands? 

6.14 

Do  data  entry  screens  and  dialog  boxes  indicate  the  number  of  character  spaces  available  in  a 
field? 

6.15 

Do  fields  in  data  entry  screens  and  dialog  boxes  contain  default  values  when  appropriate? 

7.  Recognition  Rather  Than  Recall 

Minimize  the  user's  memory  load  by  making  objects,  actions,  and  options  visible. 
The  user  should  not  have  to  remember  information  from  one  part  of  the  dialogue  to 
another.  Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable 
whenever  appropriate. 


# 

Review  Checklist 

7.1 

For  question  and  answer  interfaces,  are  visual  cues  and  white  space  used  to  distinguish 
questions,  prompts,  instructions,  and  user  input? 

7.2 

Does  the  data  display  start  in  the  upper-left  corner  of  the  screen? 

7.3 

Are  multiword  field  labels  placed  horizontally  (not  stacked  vertically)? 

7.4 

Are  all  data  a  user  needs  on  display  at  each  step  in  a  transaction  sequence? 

7.5 

Are  prompts,  cues,  and  messages  placed  where  the  eye  is  likely  to  be  looking  on  the  screen? 

7.6 

Have  prompts  been  formatted  using  white  space,  justification,  and  visual  cues  for  easy 
scanning? 

7.7 

Do  text  areas  have  "breathing  space"  around  them? 

7.8 

Is  there  an  obvious  visual  distinction  made  between  "choose  one"  menu  and  "choose  many" 
menus? 

7.9 

Have  spatial  relationships  between  soft  function  keys  (on-screen  cues)  and  keyboard  function 
keys  been  preserved? 

7.10 

Does  the  system  gray  out  or  delete  labels  of  currently  inactive  soft  function  keys? 

7.11 

Is  white  space  used  to  create  symmetry  and  lead  the  eye  in  the  appropriate  direction? 
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7.12 

Have  items  been  grouped  into  logical  zones,  and  have  headings  been  used  to  distinguish 
between  zones? 

7.13 

Are  zones  no  more  than  twelve  to  fourteen  characters  wide  and  six  to  seven  lines  high? 

7.14 

Have  zones  been  separated  by  spaces,  lines,  color,  letters,  bold  titles,  rules  lines,  or  shaded 
areas? 

7.15 

Are  field  labels  close  to  fields,  but  separated  by  at  least  one  space? 

7.16 

Are  long  columnar  fields  broken  up  into  groups  of  five,  separated  by  a  blank  line? 

7.17 

Are  optional  data  entry  fields  clearly  marked? 

7.18 

Are  symbols  used  to  break  long  input  strings  into  "chunks"? 

7.19 

Is  reverse  video  or  color  highlighting  used  to  get  the  user's  attention? 

7.20 

Is  reverse  video  used  to  indicate  that  an  item  has  been  selected? 

7.21 

Are  size,  boldface,  underlining,  color,  shading,  or  typography  used  to  show  relative  quantity  or 
importance  of  different  screen  items? 

7.22 

Are  borders  used  to  identify  meaningful  groups? 

7.23 

Has  the  same  color  been  used  to  group  related  elements? 

7.24 

Is  color  coding  consistent  throughout  the  system? 

7.25 

Is  color  used  in  conjunction  with  some  other  redundant  cue? 

7.26 

Is  there  good  color  and  brightness  contrast  between  image  and  background  colors? 

7.27 

Have  light,  bright,  saturated  colors  been  used  to  emphasize  data  and  have  darker,  duller,  and 
de-saturated  colors  been  used  to  de-emphasize  data? 

7.28 

Is  the  first  word  of  each  menu  choice  the  most  important? 

7.29 

Does  the  system  provide  mapping:  that  is,  are  the  relationships  between  controls  and  actions 
apparent  to  the  user? 

7.30 

Are  input  data  codes  distinctive? 

7.31 

Have  frequently  confused  data  pairs  been  eliminated  whenever  possible? 

7.32 

Have  large  strings  of  numbers  or  letters  been  broken  into  chunks? 

7.33 

Are  inactive  menu  items  grayed  out  or  omitted? 

7.34 

Are  there  menu  selection  defaults? 

7.35 

If  the  system  has  many  menu  levels  or  complex  menu  levels,  do  users  have  access  to  an  on¬ 
line  spatial  menu  map? 

7.36 

Do  GUI  menus  offer  affordance:  that  is,  make  obvious  where  selection  is  possible? 

7.37 

Are  there  salient  visual  cues  to  identify  the  active  window? 

7.38 

Are  function  keys  arranged  in  logical  groups? 

7.39 

Do  data  entry  screens  and  dialog  boxes  indicate  when  fields  are  optional? 

7.40 

On  data  entry  screens  and  dialog  boxes,  are  dependent  fields  displayed  only  when  necessary? 

8.  Flexibility  and  Efficiency  of  Use 

Accelerators,  unseen  by  the  novice  user,  may  often  speed  up  the  interaction  for 
the  expert  user  such  that  the  system  can  cater  to  both  inexperienced  and  experienced 
users.  Allow  users  to  tailor  frequent  actions.  Provide  alternative  means  of  access  and 
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operation  for  users  who  differ  from  the  “average”  user  (e.g.,  physical  or  cognitive  ability, 
culture,  language,  etc). 


# 

Review  Checklist 

8.1 

If  the  system  supports  both  novice  and  expert  users,  are  multiple  levels  of  error  message  detail 
available? 

8.2 

Does  the  system  allow  novices  to  use  a  keyword  grammar  and  experts  to  use  a  positional 
grammar? 

8.3 

Can  users  define  their  own  synonyms  for  commands? 

8.4 

Does  the  system  allow  novice  users  to  enter  the  simplest,  most  common  form  of  each 
command,  and  allow  expert  users  to  add  parameters? 

8.5 

Do  expert  users  have  the  option  of  entering  multiple  commands  in  a  single  string? 

8.6 

Does  the  system  provide  function  keys  for  high-frequency  commands? 

8.7 

For  data  entry  screens  with  many  fields  or  in  which  source  documents  may  be  incomplete,  can 
users  save  a  partially  filled  screen? 

8.8 

Does  the  system  automatically  enter  leading  zeros? 

8.9 

If  menu  lists  are  short  (seven  items  or  fewer),  can  users  select  an  item  by  moving  the  cursor? 

8.10 

If  the  system  uses  a  type-ahead  strategy,  do  the  menu  items  have  mnemonic  codes? 

8.11 

If  the  system  uses  a  pointing  device,  do  users  have  the  option  of  either  clicking  on  fields  or 
using  a  keyboard  shortcut? 

8.12 

Does  the  system  offer  "find  next"  and  "find  previous"  shortcuts  for  database  searches? 

8.13 

On  data  entry  screens,  do  users  have  the  option  of  either  clicking  directly  on  a  field  or  using  a 
keyboard  shortcut? 

8.14 

On  menus,  do  users  have  the  option  of  either  clicking  directly  on  a  menu  item  or  using  a 
keyboard  shortcut? 

8.15 

In  dialog  boxes,  do  users  have  the  option  of  either  clicking  directly  on  a  dialog  box  option  or 
using  a  keyboard  shortcut? 

8.16 

Can  expert  users  bypass  nested  dialog  boxes  with  either  type-ahead,  user-defined  macros,  or 
keyboard  shortcuts? 

9.  Aesthetic  and  Minimalist  Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed. 
Every  extra  unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of 
information  and  diminishes  their  relative  visibility. 


# 

Review  Checklist 

9.1 

Is  only  (and  all)  information  essential  to  decision  making  displayed  on  the  screen? 

9.2 

Are  all  icons  in  a  set  visually  and  conceptually  distinct? 

9.3 

Have  large  objects,  bold  lines,  and  simple  areas  been  used  to  distinguish  icons? 
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9.4 

Does  each  icon  stand  out  from  its  background? 

9.5 

If  the  system  uses  a  standard  GUI  interface  where  menu  sequence  has  already  been  specified, 
do  menus  adhere  to  the  specification  whenever  possible? 

9.6 

Are  meaningful  groups  of  items  separated  by  white  space? 

9.7 

Does  each  data  entry  screen  have  a  short,  simple,  clear,  distinctive  title? 

9.8 

Are  field  labels  brief,  familiar,  and  descriptive? 

9.9 

Are  prompts  expressed  in  the  affirmative,  and  do  they  use  the  active  voice? 

9.10 

Is  each  lower-level  menu  choice  associated  with  only  one  higher  level  menu? 

9.11 

Are  menu  titles  brief,  yet  long  enough  to  communicate? 

9.12 

Are  there  pop-up  or  pull-down  menus  within  data  entry  fields  that  have  many,  but  well- 
defined,  entry  options? 

10.  Help  and  Documentation 

Even  though  it  is  better  if  the  system  can  be  used  without  documentation,  it  may 
be  necessary  to  provide  help  and  documentation.  Any  such  information  should  be  easy  to 
search,  focused  on  the  user's  task,  list  concrete  steps  to  be  carried  out,  and  not  be  too 
large. 


# 

Review  Checklist 

10.1 

If  users  are  working  from  hard  copy,  are  the  parts  of  the  hard  copy  that  go  on-line  marked? 

10.2 

Are  on-line  instructions  visually  distinct? 

10.3 

Do  the  instructions  follow  the  sequence  of  user  actions? 

10.4 

If  menu  choices  are  ambiguous,  does  the  system  provide  additional  explanatory  information 
when  an  item  is  selected? 

10.5 

Are  data  entry  screens  and  dialog  boxes  supported  by  navigation  and  completion  instructions? 

10.6 

If  menu  items  are  ambiguous,  does  the  system  provide  additional  explanatory  information 
when  an  item  is  selected? 

10.7 

Are  there  memory  aids  for  commands,  either  through  on-line  quick  reference  or  prompting? 

10.8 

Is  the  help  function  visible;  for  example,  a  key  labeled  HELP  or  a  special  menu? 

10.9 

Is  the  help  system  interface  (navigation,  presentation,  and  conversation)  consistent  with  the 
navigation,  presentation,  and  conversation  interfaces  of  the  application  it  supports? 

10.10 

Navigation:  Is  information  easy  to  find? 

10.11 

Presentation:  Is  the  visual  layout  well  designed? 

10.12 

Conversation:  Is  the  information  accurate,  complete,  and  understandable? 

10.13 

Is  the  information  relevant? 

10.14 

Goal-oriented  (What  can  I  do  with  this  program?) 

10.15 

Descriptive  (What  is  this  thing  for?) 

10.16 

Procedural  (How  do  I  do  this  task?) 

10.17 

Interpretive  (Why  did  that  happen?) 
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10.18 

Navigational  (Where  am  I?) 

10.19 

Is  there  context-sensitive  help? 

10.20 

Can  the  user  change  the  level  of  detail  available? 

10.21 

Can  users  easily  switch  between  help  and  their  work? 

10.22 

Is  it  easy  to  access  and  return  from  the  help  system? 

10.23 

Can  users  resume  work  where  they  left  off  after  accessing  help? 

11.  Skills 

The  system  should  support,  extend,  supplement,  or  enhance  the  user’s  skills, 
background  knowledge,  and  expertise  —  not  replace  them. 


# 

Review  Checklist 

11.1 

Can  users  choose  between  iconic  and  text  display  of  information? 

11.2 

Are  window  operations  easy  to  learn  and  use? 

11.3 

If  users  are  experts,  usage  is  frequent,  or  the  system  has  a  slow  response  time,  are  there  fewer 
screens  (more  information  per  screen)? 

11.4 

If  users  are  novices,  usage  is  infrequent,  or  the  system  has  a  fast  response  time,  are  there  more 
screens  (less  information  per  screen)? 

11.5 

Does  the  system  automatically  color-code  items,  with  little  or  no  user  effort? 

11.6 

If  the  system  supports  both  novice  and  expert  users,  are  multiple  levels  of  detail  available. 

11.7 

Are  users  the  initiators  of  actions  rather  than  the  responders? 

11.8 

Does  the  system  perform  data  translations  for  users? 

11.9 

Do  field  values  avoid  mixing  alpha  and  numeric  characters  whenever  possible? 

11.10 

If  the  system  has  deep  (multilevel)  menus,  do  users  have  the  option  of  typing  ahead? 

11.12 

When  the  user  enters  a  screen  or  dialog  box,  is  the  cursor  already  positioned  in  the  field  users 
are  most  likely  to  need? 

11.13 

Can  users  move  forward  and  backward  within  a  field? 

11.14 

Is  the  method  for  moving  the  cursor  to  the  next  or  previous  field  both  simple  and  visible? 

11.15 

Has  auto-tabbing  been  avoided  except  when  fields  have  fixed  lengths  or  users  are 
experienced? 

11.16 

Do  the  selected  input  device(s)  match  user  capabilities? 

11.17 

Are  cursor  keys  arranged  in  either  an  inverted  T  (best  for  experts)  or  a  cross  configuration 
(best  for  novices)? 

11.18 

Are  important  kevs  (for  example,  ENTER  ,  TAB)  larger  than  other  kevs? 

11.19 

Are  there  enough  function  keys  to  support  functionality,  but  not  so  many  that  scanning  and 
finding  are  difficult? 

11.20 

Are  function  keys  reserved  for  generic,  high-frequency,  important  functions? 

11.21 

Are  function  key  assignments  consistent  across  screens,  subsystems,  and  related  products? 

11.22 

Does  the  system  correctly  anticipate  and  prompt  for  the  user's  probable  next  activity? 
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12.  Pleasurable  and  Respectful  Interaction  with  the  User 


The  user’s  interactions  with  the  system  should  enhance  the  quality  of  her  or  his 
work-life.  The  user  should  be  treated  with  respect.  The  design  should  be  aesthetically 
pleasing  with  artistic  as  well  as  functional  value. 


# 

Review  Checklist 

12.1 

Is  each  individual  icon  a  harmonious  member  of  a  family  of  icons? 

12.2 

Has  excessive  detail  in  icon  design  been  avoided? 

12.3 

Has  color  been  used  with  discretion? 

12.4 

Has  the  amount  of  required  window  housekeeping  been  kept  to  a  minimum? 

12.5 

If  users  are  working  from  hard  copy,  does  the  screen  layout  match  the  paper  form? 

12.6 

Has  color  been  used  specifically  to  draw  attention,  communicate  organization,  indicate  status 
changes,  and  establish  relationships? 

12.7 

Can  users  turn  off  automatic  color  coding  if  necessary? 

12.8 

Are  typing  requirements  minimal  for  question  and  answer  interfaces? 

12.9 

Do  the  selected  input  device(s)  match  environmental  constraints? 

12.13 

If  the  system  uses  multiple  input  devices,  has  hand  and  eye  movement  between  input  devices 
been  minimized? 

12.14 

If  the  system  supports  graphical  tasks,  has  an  alternative  pointing  device  been  provided? 

12.15 

Is  the  numeric  keypad  located  to  the  right  of  the  alpha  key  area? 

12.16 

Are  the  most  frequently  used  function  keys  in  the  most  accessible  positions? 

12.17 

Does  the  system  complete  unambiguous  partial  input  on  a  data  entry  field? 

13.  Privacy 

The  system  should  help  the  user  to  protect  personal  or  private  information 
belonging  to  the  user  or  his/her  clients. 


# 

Review  Checklist 

13.1 

Are  protected  areas  completely  inaccessible? 

13.2 

Can  protected  or  confidential  areas  be  accessed  with  certain  passwords? 

13.3 

Is  this  feature  effective  and  successful? 
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Appendix  B.  Summary  of  Military  Guidance  Documents 


The  following  information  summarizes  the  findings  of  the  review  of  military 
guidance.  The  first  bullet  for  each  document  describes  its  relationship  with  web  site 
development,  maintenance,  and  use.  The  remaining  bullets  identify  specific  the  usability 
heuristic  and  Section  508  references  contained  within  each  document.  Reference 
paragraphs  are  provided  where  applicable. 


Air  Force  Guidance 

Air  Force  Instruction  33-117 
Multimedia  (MM)  Management 
1  April  2004 

Directly  applicable  to  multimedia  productions,  including  those  developed  for 
web  site  use,  by  military  MM  functions. 

General  Section  508  compliance  statement s) 

Reference  paragraphs(s) :  1.1 


Air  Force  Instruction  33-129 
Web  Management  and  Internet  Use 
3  February  2005 

Directly  applicable  to  web  site  design,  maintenance,  and  use 

General  Section  508  compliance  statement s) 

Reference  paragraph(s):  3.5.1,  3. 9. 1.3,  3.12.8,  3.14.1,  3.14.8 

Consistency  and  standards 
Reference  paragraphs(s) :  3.13.1 

Pleasurable  and  respectful  interaction  with  the  user 
Reference  paragraphs(s):  6 

Aesthetic  and  minimalist  design 
Reference  paragraphs(s):  6 
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Privacy 

Reference  paragraphs(s):  Paragraphs  3. 9. 1.3,  3.12.8,  3.12.1,  3.13.1,  3.15.3, 
5.1.2,  7.1.4.9,  7.2,  12.1.1,  12.2,  12.6 

Note:  references  IEEE  Standard  2001  as  an  additional  information  source,  but 
does  not  mandate  compliance  with  the  principles  contained  within  that 
document. 


Air  Force  Instruction  33-219 

Telecommunications  Monitoring  and  Assessment  Program 
23  May  2002 

Does  not  specifically  address  web  site  development,  maintenance,  or  use  but 
is  applicable  to  web  sites  because  they  are  a  form  of  telecommunications  that 
fall  under  the  scope  of  the  program  established  by  this  document.  At 
minimum,  it  is  an  additional  mandatory  reference. 

Privacy 

Reference  paragraphs(s):  21.8,  A3. 3. 6 


Air  Force  Instruction  33-332 
Privacy  Act  Program 
29  January  2004 

Contains  multiple  comments  directed  specifically  at  web  site  development, 
maintenance,  and  use. 

Privacy 

Reference  paragraphs(s):  12.8,  12.8.1,  12.8.2,  12.8.3 


Air  Force  Instruction  35-101 
Public  Affairs  Policies  and  Procedures 
29  November  2005* 

Directly  applicable  to  web  site  design,  maintenance,  and  use.  Provides 
guidance  for  both  public  and  private  web  sites  but  only  those  paragraphs 
applicable  to  private  web  sites  are  evaluated  for  this  study 


Previous  version  not  available  for  review. 
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Privacy 

Reference  paragraphs(s): 

-  Section  508  §  1194.22(a) 
Reference  paragraphs(s): 

-  Section  508  §  1194.22(b) 
Reference  paragraphs(s): 

-  Section  508  §  1194.22(c) 
Reference  paragraphs(s): 

-  Section  508  §  1194.22(d) 
Reference  paragraphs(s): 

-  Section  508  §  1194.22(e) 
Reference  paragraphs(s): 

-  Section  508  §  1194.22(g) 
Reference  paragraphs(s): 


18.5.3,  18.8.1.3,  10.10 

18.7.1.1,  18.7.1.7 

18.7.1.2,  18.7.1.8 

18.7.1.3 

18.7.1.4 

18.7.1.5 

18.7.1.6 


Air  Force  Manual  33-326 
Preparing  Official  Communications 
1  November  1999 

Does  not  specifically  address  web  site  development,  maintenance,  or  use  but 
could  be  construed  as  applicable  to  web  sites  that  function  as  a  form  of  official 
communication.  At  minimum,  it  is  an  additional  mandatory  reference. 

Match  between  the  system  and  the  real  world 
Reference  paragraphs(s):  1.2 


Air  Force  Manual  37-104 

Managing  Information  to  Support  the  Air  Force  Mission 
1  June  1995 

Does  not  specifically  address  web  site  development,  maintenance,  or  use  but 
is  applicable  to  web  sites  because  they  are  a  form  of  information  management. 
At  minimum,  it  is  an  additional  mandatory  reference. 

Privacy 

Reference  paragraphs(s):  4.1,  4.2 
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Air  Force  Policy  Directive  37-1 
Information  Management 
19  November  1993 


Does  not  specifically  address  web  site  development,  maintenance,  or  use  but 
could  be  construed  as  applicable  to  web  sites  that  function  as  a  form  of  official 
communication.  At  minimum,  it  is  an  additional  mandatory  reference. 


Privacy 

Reference  paragraphs(s):  2.6 


Air  Force  Supplement  to  Department  of  Defense  Regulation  5400.7 
DoD  Freedom  of  Information  Act  Program 
24  June  2002 

Contains  one  comment  directed  specifically  at  web  site  development, 
maintenance,  and  use. 

Privacy 

Reference  paragraph(s):  C3.2.1.6 


Air  Force  Content  Publishing  Guide  for  Publishing  Center  v  1 . 1 
April  2004 


Addresses  web  site  design  and  maintenance  for  sites  associated  with  the  Air 
Force  Portal.  Publication  is  available  for  other  uses,  but  compliance  with  it  is 
not  mandatory  for  those  uses. 

Guide  discusses  general  requirements  for  content  provided  by  Portal  web 
sites,  but  overall  themes  include: 

-  Visibility  of  System  Status 

-  Match  between  system  and  the  real  world 

-  User  control  and  freedom 

-  Consistency  and  standards 

-  Recognition  rather  than  recall 

-  Flexibility  and  efficiency  of  use 

-  Aesthetic  and  minimalist  design 

Guide  discusses  specific  requirement  to  meet  Section  508  standards  and 
explicitly  provides  general  guidance  for  the  following  Section  508  1 194.22 
standards: 

-  Section  508  §  1194.22(a) 

-  Section  508  §  1194.22(b) 


92 


-  Section  508  §  1194.22(f) 

-  Section  508  §  1194.22(h) 

-  Section  508  §1194.22(i) 

-  Section  508  §1194.22(1) 


Air  Force  Portal  Style  Guide  v  2.0 
October  2003 

Addresses  web  site  design  and  maintenance  for  sites  associated  with  the  Air 
Force  Portal.  Publication  is  available  for  other  uses,  but  compliance  with  it  is 
not  mandatory  for  those  uses. 

Guide  discusses  specific  requirements  for  Portal  web  sites,  but  overall  themes 
include: 

Visibility  of  system  status 

Consistency  and  standards 

Recognition  rather  than  recall 

Pleasurable  and  respectful  interaction  with  the  user 

Department  of  Defense  Guidance 

Department  of  Defense  Directive  Number  8000.1 

Management  of  DoD  Information  Resources  and  Information  Technology 
27  February  2002 

Does  not  specifically  address  web  site  development,  maintenance,  or  use 
but  is  applicable  to  web  sites  because  they  are  a  form  of  Information 
Resources  and  Infonnation  Technology  to  which  this  directive  is 
applicable.  At  minimum,  it  is  an  additional  mandatory  reference. 

General  Section  508  compliance  statement s) 

Reference  paragraphs(s) :  4.4.3,4.10 


Office  of  the  Assistant  Secretary  of  Defense 

Web  Site  Administration  Policies  &  Procedures 

November  25,  1998  (latest  corrections  from  1 1  January,  2002) 

Directly  applicable  to  web  site  design,  maintenance,  and  use  although 
focused  more  on  web  site  content  than  on  site  design. 

Privacy 

Reference  paragraph(s):  Part  I  4.2,  4.4,  5.5.4,  5.5.10;  Part  II  3.6,  11.1, 
12.3 
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